3

I started out looking for a way to test the difference between MSPE between two models (Question here), when (thanks to @Richard Hardy) I ended up reading a paper of Diebold regarding the Diebold-Mariano test (Comparing Predictive Accuracy, Twenty Years Later: A Personal Perspective on the Use and Abuse of Diebold-Mariano Tests).

Diebold claims two things that I find curious:

... the errors are driven by forecasts, not models

and

The DM test was intended for comparing forecasts; it has been, and remains, useful in that regard. The DM test was not intended for comparing models.

Since models are doing the forecasting, how come errors are not driven by models?
And if the DM test compares forecasts (generated by the models), how come this is not effectively comparing models?

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
Erosennin
  • 1,384
  • 17
  • 31

1 Answers1

2

What Diebold does not seem to talk about directly in his paper but what might be the answer to your question is the following. There are two different questions/problems:

  1. Which forecast is better (not only in this particular sample, but in population)? vs.
  2. Which model is more likely to have generated the data?

You could have Model A that generated the data but that is difficult to estimate with high precision. Once estimated, Model A produces poor forecasts due to the estimation imprecision.
You could have Model B that did not generate the data but approximates the data quite well and is also easy to estimate with high precision. Once estimated, Model B delivers rather accurate forecasts (more accurate than those from Model A).
Forecasts generated by Model B is the answer to question 1, while Model A is the answer to question 2.

See also my answer in the thread "Tests of Forecast Accuracy for Nested Models".

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
  • Aha, I think I get it - at least on a higher level. Thank you for this! So we think of models as generators of data, and forecasts as results of our "forecasting models"? – Erosennin Jun 26 '17 at 13:55
  • 1
    @Erosennin, This might be too general a statement in a broader context, but here it might be right. I am not sure if the answer addresses the real cause of the problem in the Diebold-Mariano case, but the phenomenon I describe certainly exists and can cause similar problems. So I think it is a good candidate for explaining what Diebold had in mind. – Richard Hardy Jun 26 '17 at 14:41
  • 1
    @Erosennin, Thanks! I really like this type of questions and topics, and I hope I am correct on this one. There are ideas like this that you can lay out in just a few sentences, but boy what a difference they make! – Richard Hardy Jun 26 '17 at 14:45
  • I agree very much :) A few sentences can make such an impact :) – Erosennin Jun 28 '17 at 18:54