1

Now that I know why it is important to represent the uncertainty of a model, I would like to know how to best represent uncertainty in a time series forecasting model.

My previous question introduced me to many different kinds of uncertainty measures. Confidence Intervals, Prediction Intervals, Credible Intervals and Standard Errors (there are probably more which weren't covered).

I've been asked to use Confidence Intervals, but every time I read about them, Confidence Intervals are mentioned in the context of the Gaussian Normal distribution (bell shaped). My time series data does not follow the bell shape.

I also read a discussion on the differences between the Confidence and Prediction intervals. It says that Confidence Intervals are narrower than Prediction Intervals, but I'm not sure how this helps me choose which one to use.

My question is: now that I have made some point forecasts on a time series using my model, how do I choose the most appropriate method to measure my model's uncertainty?

Marcus
  • 255
  • 1
  • 4

2 Answers2

2

The key difference between CIs and PIs is not that the first are narrower, but that they encode the uncertainty in two different things.

  • CIs give the uncertainty in unobservable parameter estimates. (And they indeed are often derived from non-Gaussian distributions, e.g., the CI for a Bernoulli parameter which is close to 0 or 1. And indeed, the Gaussianity of the observed data has almost nothing to do with whether your parameter estimates are normally distributed.)

  • PIs give the uncertainty in future observable actuals.

Two very different things.

So the question really is what you are more interested in: the uncertainty in some underlying parameter (which one?)? Then use CIs. This could actually also be the uncertainty in the future conditional expected value of your time series. But remember that you will never observe this expectation. All you will observe is future actuals, which are the unknown expectation plus some residual variance. Or are you more interested in quantifying the uncertainty in future observables? Then use PIs.

I have to say that in 15 years of work in time series and forecasting, I have extremely rarely (if ever) encountered any interest in CIs, whereas PIs are all over the place (as they should be). As such, I would recommend you talk to whoever asked you to report CIs and figure out whether what they are really interested in is PIs, which seems very likely to me.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
1

An important point to me from the above is that CI do not have to assume normality. They can assume any distribution. Also, having never heard of PI before this post which is at best distressing after all the years I read time series, I don't think either CI or PI get stressed that much especially by practitioners (although unlike Stephen Kolassa I am not an expert I just read the literature and run models). I think uncertainty for time series is pretty large, almost certainly greater than a CI. For one thing structural breaks are a real issue. Ask anyone plotting financial data what COVID19 did to their models. I know a state agency that has on their wall their ARIMA models before the 2008 crash... and what really occurred. So uncertainty in the future is very high and probably unknowable. I doubt you can ever calculate it ahead of time, the best you can do is sensitivity models. What if it is ten percent, twenty percent etc higher. What if this happens...

user54285
  • 751
  • 4
  • 9