0

I have a VARX model with 3 endogenous variables in 2nd differences. Each of 3 equations has very low $R^2$ (about 0,02), but the model gives good forecast (MAPEs are about 2%).

Can I neglect low $R^2$ in this case? What is the usual practice of using $R^2$ in time series models? Does this metric really capture the accuracy of such models?

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
Elena
  • 3
  • 2
  • 1
    See also ["Why not using the R squared to measure forecast accuracy?"](https://stats.stackexchange.com/questions/195387/why-not-using-the-r-squared-to-measure-forecast-accuracy) and [R-squared to compare forecasting techniques](https://stats.stackexchange.com/questions/99899/r-squared-to-compare-forecasting-techniques/99908#99908). – Richard Hardy Apr 30 '18 at 07:31

1 Answers1

2

$R^2$ is a goodness-of-fit measure; it measures how well the model fits the data that you have. What it does not do, is measure how well the model will fit the data that you don't have yet (i.e. how well it will forecast). In fact, you can trivially build a model for any dataset that will have $R^2=1$. This will be a large model which explains every bit of noise in your data, but it will not generalize to the future: forecasts will be poor. Bottom line is: if you want to forecast, look at forecast accuracy metrics, not $R^2$.

For your particular case, I would consider reducing the model and re-checking the MAPEs because it may improve your forecasts. I would also double-check the evidence for 2nd differences because that is somewhat unusual, depending on the subject domain.

Chris Haug
  • 4,893
  • 1
  • 17
  • 24
  • 1
    Chris Haug, thank you so mush for answer! "if you want to forecast, look at forecast accuracy metrics, not R2" - new idea for me, in theory the higher R2 -> the better predictive power of a model -> the better forecast. Can you give an example of some papers about it? – Elena Mar 16 '17 at 13:19
  • "I would consider reducing the model and re-checking the MAPEs because it may improve your forecasts" - unfortunately, I can't reduce the quantity of endogenous variables, because it's important in this case. – Elena Mar 16 '17 at 13:19
  • "I would also double-check the evidence for 2nd differences because that is somewhat unusual, depending on the subject domain" - the 2st lag has better MAPE for this model (compared the 1st lag) – Elena Mar 16 '17 at 13:20
  • @Elena Sorry that wasn't clear, by "reducing" I meant setting some of the coefficients to zero, not removing endogenous variables. – Chris Haug Mar 16 '17 at 13:48