How to evaluate uncertainty estimates in regression?

Question

Some regression algorithms (e.g. Gaussian process regression) can produce uncertainties along with point predictions at test time.

These should also be evaluated. How about calculating the Pearson correlation between the standard deviation as predicted by the regression model vs. actual absolute error? Conceptually what I mean is that you'd make a scatter plot of the true absolute errors at each test point vs the predicted uncertainty of the model at those test points.

This is just a quick first idea. Are there other usual methods for evaluating the quality of predicted uncertainties?

Answer 1

In regression, there are two different kinds of "uncertainties", which correspond to the difference (and frequently, confusion) between confidence intervals and prediction intervals:

• You could have uncertainties in parameter estimates. These are captured in confidence-intervals. These, in turn, are hard to evaluate, because we usually do not know the true values of parameters. Thus, the most we can usually do is theoretical quality guarantees of CIs, frequently only in large sample limits.

• You could have uncertainties in future observables, which are captured in prediction-intervals or more generally in predictive densities. These can be evaluated much more easily, because we will actually observe those observables at some point.

  This previous thread discusses the evaluation of prediction intervals - the most common tool is the interval score. Predictive densities, in turn, are evaluated using proper scoring rules.

  More information and pointers to literature can be found in Petropoulos et al. (2021), Forecasting: theory and practice (recently accepted for publication in the International Journal of Forecasting), specifically in section 2.12.2 for prediction intervals (full disclosure: I wrote that section) and in section 2.12.4 by Florian Ziel on predictive densities.