0

I developed many forecasting models over the same dataset (multiple iteration of simulated time series data). My dataset basically is a multivariate timeseries so the forecasting models forecast many values at the same time. My question is how can I decide which forecasting model is better. I already have calculated the MSE and MAE for each model, but I want to compare them statically. I read about Diebold Mariano test but I think it used for comparing forecasts not models. I also thought of t-test for dependent data but am not sure how can I apply it in my case where I have many models (more than two)

I appreciate your help.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Manal
  • 11
  • 3
  • If your end goal is forecasting, why isn't it enough to compare the models' forecasts? [Models can differ and still give forecasts that are comparably good.](https://stats.stackexchange.com/q/459073/1352) – Stephan Kolassa Apr 08 '20 at 11:44
  • Thank you for your prompt reply. I am just wondering that if I had 10 iterations of the simulation which create slightly different times-series datasets, then I apply different forecasting models (let's say 15 machine-learning models) on these datasets then I took the average of forecasting errors for each model. How can I compare them statically? I know that Diebold Mariano deals with raw errors not (MSE or MAE) In my case, the performance of each model (15 models) represents the average of performance on 10 datasets (and the performance itself represents multiple outputs) – Manal Apr 08 '20 at 13:28

0 Answers0