1

I'm comparing different ARMA-GARCH specification out-of-sample in order to understand whether the more "parsimonious" models prescribed by BIC do not perform more poorly than the more "expensive" ones imposed by AIC.

Let's say for one of them I got a MSE=0.0031 for the MA(1), while MSE=0.0032 for ARMA(4,5).

Questions:

  • How can I say whether MSE=0.0031 is good or not? What does it mean practically (is the market inefficient)?
  • Is the difference from MA(1) and ARMA(4,5) negligible or not? Is model 1 preferable? Why or why not?
toyo10
  • 71
  • 1
  • 10
  • 1. You can't, based on the information given. You can make MSE as small as you want by dividing all the data by a sufficiently large constant. 2. If you are adding eight parameters to a one-parameter model and seeing MSE increase, you are doing something wrong. Note that the MA(1) model is nested in the ARMA(4,5) model. It cannot be that you are adding eight parameters and seeing MSE go up, unless you have an objective other than minimizing MSE - in which case, you should be reporting that objective value, not MSE. – jbowman Jul 08 '18 at 22:18
  • @jbowman: regarding (2), I assume that the MSEs reported are out-of-sample, in which case the larger model can definitely have a larger MSE than a nested smaller one through overfitting. – Stephan Kolassa Jul 09 '18 at 06:16
  • @StephanKolassa - which goes to show I shouldn't try to read before I've had my morning espresso... – jbowman Jul 09 '18 at 21:34

1 Answers1

2

  1. As @jbowman notes, you can't say whether a given MSE is "good" or "bad". You can say whether one method is better than another one. It is always good to compare your forecasts to a simple benchmark. For instance, the simple historical mean can be very hard to beat.

  2. You can test the difference between two forecasting errors for statistical significance. The most common test is the test.

That said, ARMA(4,5) is a huge model. It's already hard to find real-life examples of MA processes. I am skeptical anything can be meaningfully described with not one, but five MA terms. (I expect some people will disagree.)

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357