Optimality of AIC w.r.t. loss functions used for evaluation

Question

Under certain conditions, AIC is an efficient model selection criterion. I understand this roughly as if AIC will tend to select the model that will yield the largest expected likelihood of a new data point from the same data generating process or population (among all models that we are selecting from). This makes AIC the preferred choice if the goal is prediction and the evaluation of predictions is the likelihood.

However, we do not always evaluate prediction accuracy by the likelihood. There are other means of evaluating predictions such as, say, mean squared error (MSE) or mean absolute error (MAE). Questions:

1. Is AIC still the model selection method of choice if prediction accuracy is evaluated by these loss functions (MSE, MAE)?
2. What could be a good counterexample, preferably among the well-known loss functions? I.e. what loss function would not favor AIC as the model selection criterion?
3. How can we characterize the entirety of loss functions for evaluating prediction accuracy that are compatible with AIC being the method of choice for model selection?

Answer 1

I think the answer to 1) should be "no", as there is no reason in general to expect that the model which minimizes the expected likelihood will also minimize the MSE, MAE, etc. One might even think of the case in which the likelihood is well defined and the MSE will diverge as the sample size increases (e.g. a distribution with no moments, such as the Cauchy).

I would think that AIC will still be a good method for model selection for any loss function which is a monotone function of expected likelihood, but that's probably a trivial remark that will not help you much.

Answer 2

I have to disagree with F. Tusell's answer, which I believe reflects a confusion about what the AIC and other loss functions evaluate.

The AIC evaluates a "modeling" density. (I use quotes around "modeling", to distinguish it from a predictive density, where we would use proper scoring rules for evaluation.) Loss functions like the MAE, the MSE and quantile losses evaluate single number summaries (Kolassa, 2020, IJF) of such modeling (or predictive) densities.

Now, by Stone (1977), the AIC will asymptotically be minimized by the true conditional density (provided it is in the candidate pool; more on this below). Once we have the true conditional density, we can extract the functional from it that minimizes the loss function (the conditional expectation for the MSE, the median for the MAE, the quantile for the quantile loss). Thus, the procedure of "pick the density that minimizes AIC, then extract the appropriate functional for our loss" will asymptotically yield the lowest loss.

Now, all this of course relies on a number of assumptions.

• As F. Tusell writes, if the conditional density does not have an expectation, then the "extract the minimum MSE functional" part will not work, so the entire pipeline breaks down. (But if the true DGP follows a Cauchy distribution, what would be the optimal point prediction under the MSE, anyway?)

• If the true conditional density is not in our candidate pool of possible models, the asymptotics results of Stone (1977) do not hold. AIC will still asymptotically find the model in the pool with minimal Kullback-Leibler distance from the true DGP, so it would still be a good start - although that one might have a worse expected loss than some other model.

  As an example, your data may be $N(0,2)$ distributed, but our model pool may only contain all $N(\mu,1)$ distributions, so the key assumption of Stone (1977) is not satisfied. Assume we are interested in a 90% quantile prediction. The AIC will be optimized in our pool by $N(0,1)$ and report its quantile of $q_{90\%}(0,1)\approx 1.28$. A distribution-free approach that simply optimizes on quantile loss will yield the correct $1.81$. And of course, there is a model in our pool that would yield a lower loss, namely $N(0.54,1)$ with $q_{90\%}(0.54,1)\approx 1.81$, but the AIC won't like that one.

• Finally, of course if the functional form differs between fitting and prediction, all bets are off. If you get hit by a world-wide pandemic, your AIC-optimal model based on 2019 data will not be very useful for a (point) forecast for toilet paper demand in Germany in early 2020.

• And finally-finally, asymptotics may be a long way off - far enough for the DGP to indeed change, as per the previous bullet point.