3

Suppose I have two in-sample forecasts from two different non-nested models. I want to check which one produces the best forecasts. A common way is Diebold-Mariano, GiacominiWhite, ENC-T test. However, these tests were designed for out-of-sample forecasts. Would they also work in-sample?

So, in my case, I have a time-invariant model and the same model with time-varying coefficients. I want to check whether it is important to account for time-variation in coefficients. Does AIC still work then? I thought of comparing the R2 in both models, 1 minus tge ratio of the sum of the squared residuals in both models. The issue is where to draw the line. Suppose I have unconditional model R2 of 5% and conditional one of 6 or 7% (I am dealing with asset pricing predictive regressions so the R2s are often even smaller than this). How to decide that this improvement of 1 ,2 3% in the R2 is economically significant based on R2, AIC and so on based on in-sample methods?

Daniel Pinto
  • 613
  • 4
  • 15
  • Why don't you do 1-day ahead predictions by re-estimating the model daily vs the fixed parameter model predictions. Then you can do Diebold-Mariano. Make sure too keep a constant window length for your daily re-estimations. – stollenm Apr 19 '18 at 06:09
  • Daniel, see my answers [here](https://stats.stackexchange.com/questions/257799/tests-of-forecast-accuracy-for-nested-models/286983#286983) and [here](https://stats.stackexchange.com/questions/230566/diebold-mariano-test-in-case-of-nested-models-clark-mccracken-2001/230837#230837). They were written for different questions but might be somewhat relevant to this one, too. – Richard Hardy Apr 20 '18 at 16:21

1 Answers1

2

In sample, you would look at goodness-of-fit measures, such as Bayes Information Criterion (BIC), Akaike Information Criterion (AIC) or R^2.

Generally speaking, the more parameters your model has, the better you can fit the data. For this reason there it is difficult to define a "fair" way to compare in sample fit. BIC and AIC have penalty terms for the number of parameters but theses are chosen somewhat heuristically.

stollenm
  • 742
  • 4
  • 16