1

The majority of the automatic model selection algorithms like auto.arima and ets (https://robjhyndman.com/publications/automatic-forecasting/) are using information criteria (AIC, AICc or BIC) as opposed to Cross Validation for model selection.

This makes a lot of sense for automatic model selection algorithms as it's way faster and you don't need to design your CV protocol (i.e. CV splits and horizon). These ICs approximate the one-step forecast errors. But what happens if we're interested good forecast performance for longer forecast horizons e.g (monthly data with a 12-24 step horizon)?

Is CV going to significantly outperform Information Criteria for longer forecasting horizons? (to a point where it makes sense to invest the time to use CV instead)

I'm not aware of any papers/articles making this comparison, but I think it makes for interesting discussion!

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
leonidas
  • 11
  • 1
  • I think there may be something about AIC optimizing one-step-ahead but BIC optimizing several steps ahead. This connection has been made somewhere on Cross Validated, but I do not remember where exactly and am not sure how valid it is. – Richard Hardy May 14 '21 at 06:59
  • Related recent thread ["Why is AIC or BIC commonly used in model selections for time series forecasting?"](https://stats.stackexchange.com/questions/523485) and old thread ["AIC versus cross validation in time series: the small sample case"](https://stats.stackexchange.com/questions/139175). – Richard Hardy May 14 '21 at 07:02
  • Thanks @RichardHardy! the related topics you shared are very useful. No doubt sample size plays a big role here. I'm tempted to test this out on a benchmark dataset (e.g. M3 monthly) just for fun! If I get around doing this I'll post the results here as well. – leonidas May 17 '21 at 09:59
  • The series of M3 are all pretty short, so you are unlikely to get close to asymptotics there. Consider using M4 data instead. – Richard Hardy May 17 '21 at 10:41
  • Thanks! also found a reference in Forecasting: Principles and Practice regarding your comment on BIC -> https://otexts.com/fpp3/selecting-predictors.html – leonidas May 18 '21 at 16:29

0 Answers0