10

Why in literature usually the common accuracy measures like MAD, MSE, RMSE, MAPE ... are used. Why not using the $R^2$ (coefficient of determination)?

I was thinking about the difference: By using the MSE i can compare the average of the forecast. And when using $R^2$ I'll get a information about the variance.

Why the comparison of averages is mostly commonly used? Can anybody give me an hint?

Ferdi
  • 4,882
  • 7
  • 42
  • 62
NMe
  • 103
  • 1
  • 5

1 Answers1

7

In-sample $R^2$ is not a suitable measure of forecast accuracy because it does not account for overfitting. It is always possible to build a complicated model that will fit the data perfectly in sample but there are no guarantees such a model would perform decently out of sample.

Out-of-sample $R^2$, i.e. the squared correlation between the forecasts and the actual values, is deficient in that it does not account for bias in forecasts.

For example, consider realized values

$$y_{t+1},\dotsc,y_{t+m}$$

and two competing forecasts:

$$\hat{y}_{t+1},\dotsc,\hat{y}_{t+m}$$

and

$$\tilde{y}_{t+1},\dotsc,\tilde{y}_{t+m}.$$

Now assume that

$$\tilde{y}_{t+i}=c+\hat{y}_{t+i}$$

for every $i$, where $c$ is a constant. That is, the forecasts are the same except that the second one is higher by $c$. These two forecasts will generally have different MSE, MAPE etc. but the $R^2$ will be the same.

Consider an extreme case: the first forecast is perfect, i.e. $\hat{y}_{t+i}=y_{t+i}$ for every $i$. The $R^2$ of this forecast will be 1 (which is very good). However, the $R^2$ of the other forecast will also be 1 even though the forecast is biased by $c$ for every $i$.

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
  • 1
    Why would you calculate the out-of-sample $R^2$ as the squared correlation instead of an out-of-sample $1-\frac{SSE}{TSS}$? Doing an R simulation, I get your same result that $R^2$ is unchanged upon biasing the out-of-sample predictions if I calculate $R^2$ your way, but calculating $R^2$ my way gives a terrible out-of-sample value (negative, even). – Dave Dec 30 '19 at 19:00
  • @Dave, thank you for your insight. I do not see it as *my* way vs. *someone else's* way, rather as a direct extension of a definition of an in-sample $R^2$. After all, the letter R denotes correlation, and the in-sample $R^2$ is the square of the multiple correlation between the actual and the fitted values. There are other possible ways of defining in-sample $R^2$, leading to the same in-sample quantity. Meanwhile, their out-of-sample extensions are not equivalent. My answer shows that a direct extension (though not all possible ones) can be nonsensical as a measure of forecast accuracy. – Richard Hardy Dec 30 '19 at 21:07
  • @Dave, in any case, yours is a good point! I could add a note saying that not all extensions of the in-sample $R^2$ to the out-of-sample case are so ill behaved. I would have to think which of them are sensible as measures of forecast accuracy and which of them are not. – Richard Hardy Dec 30 '19 at 21:10