0

I have computed forecasts with 4 different methods, namely OLS, Elastic Net, Cubic splines in combination with Lasso, and Neural Network. All models use the same set of base variables, except cubic splines adds the 'spline terms' as well. My first question then is, are these models nested?

I am interested to compare the forecasting performance of these methods, and was thinking of using the Diebold Mariano (1995) test. Now I read in another thread that in case the models are nested, the critical values are not good anymore. Is this true? And if so, what should I do?

Rik
  • 193
  • 5
  • Hi: your models are not nested but I would read chris haug's answer at the link below. The DM test is sometimes mis-applied so I would read the 20 years later paper to make sure you are using it correctly. https://stats.stackexchange.com/questions/421216/can-i-give-continuous-rank-probability-score-crps-to-diebold-mariano-dm-test/421340#421340 – mlofton Aug 15 '19 at 11:54
  • Thanks for your answer mlofton. Are the OLS and Elastic Net really not nested? They're both linear, using the same variables except EN has an additional penalization term in the objective which shrinks the coefficients. – Rik Aug 15 '19 at 12:33
  • [This thread](https://stats.stackexchange.com/questions/230566/diebold-mariano-test-in-case-of-nested-models-clark-mccracken-2001) and the references therein might be relevant. @mlofton, just in case, [here](https://stats.stackexchange.com/search?q=%5Bdiebold-mariano%5D+2015+user%3A53690) are my answers citing the *20 years later* paper. – Richard Hardy Aug 16 '19 at 08:32
  • [This thread](https://stats.stackexchange.com/questions/257799/tests-of-forecast-accuracy-for-nested-models) might be relevant, too. – Richard Hardy Aug 16 '19 at 08:37
  • Thanks for your reply Richard. Would this mean that the asymptotic normality assumption of the test statistic still holds in the case of nested models? My main goal is to compare the forecasting performance of the various models mentioned, so in that sense I think I'm safe? – Rik Aug 16 '19 at 12:28
  • Thanks Richard for all the links. Great material. – mlofton Aug 16 '19 at 14:09
  • @Rik: I don't know if OLS and Elastic Net are considered nested ( I don't think so ) but, as Richard has pointed out, think carefully about what you want to test because the DM testing framework is pretty different from model testing. I am pretty certain that the issue of nested or not-nested is irrelevant in the case of DM testing but Richard should chime in here ( if possible. thanks ) because it's been a looooooong time since I looked at it. – mlofton Aug 16 '19 at 14:13
  • How is the progress? Did you find an answer in any of the older threads? If not, could you clarify the goal of your analysis? Whether the vanilla critical values of DM test are appropriate may depend on the exact hypothesis you wish to test. – Richard Hardy Aug 27 '19 at 13:35
  • The other threads really improved my understanding, thank you! In my analysis I predict returns with several different models. My goal is to test whether the quality of the predictions are significantly different, i.e. it is not a comparison of “fully articulated econometric models", but rather of forecasting performance. This is why I thought the method could be appropriate, I just wasn't sure whether the models are potentially nested. The models/methods I use are simple OLS, elastic net, splines and neural network (don't know if this helps in determining whether they are nested.) – Rik Aug 27 '19 at 13:54

0 Answers0