Interpretation of AIC value

Question

Typical values of AIC that I have seen for logistic models are in thousands, at least hundreds. e.g. On http://www.r-bloggers.com/how-to-perform-a-logistic-regression-in-r/ the AIC is 727.39

While it is always said that AIC should be used only to compare models, I wanted to understand what a particular AIC value means. As per the formula, $AIC= -2 \log(L)+ 2K$

Where, L = maximum likelihood from the MLE estimator, K is number of parameters

In the above example, K =8

so, with simple arithmatic:

727.9 = -2*log(L)+ 2*8
Hence, 711.39 = -2*log(L)
Hence, log (L)= 711.39/-2 = -355.695
Hence, L = exp(-355.695) = 3.3391E-155

So, if my understanding is correct, this is the likelihood of the function identified by MLE fitting the data. This seems really really really low.

What am I missing here?

Answer 1

There is no such a thing as "typical" or correct likelihood for a model. The same with AIC, that is negative log likelihood penalized for a number of parameters. Lower value of AIC suggests "better" model, but it is a relative measure of model fit. It is used for model selection, i.e. it lets you to compare different models estimated on the same dataset.

Recall G.E.P. Box saying that "all models are wrong, but some are useful", you are not interested in finding model that has a perfect fit to your data because it is impossible and such model in many cases would be a very poor, overfitted one. Instead, you are looking for the best one that you can get, the most useful one. The general idea behind AIC is that model with lower number of parameters is better, what is somehow consistent with Occam's razor argument, that we prefer simple model over a complicated one.

You can check the following papers:

Anderson, D., & Burnham, K. (2006). AIC myths and misunderstandings.

Burnham, K.P., & Anderson, D.R. (2004). Multimodel Inference. Understanding AIC and BIC in Model Selection. Sociological Methods & Research, 33(2), 261-304.

and those threads:

What is the difference between "likelihood" and "probability"?

Is there any reason to prefer the AIC or BIC over the other?

Answer 2

AIC is highly related to generalized ("pseudo") $R^2$. I like to state AIC on the likelihood ratio $\chi^2$ scale although this is not traditional, i.e., restated AIC = $\chi^{2} - 2\times$ d.f. One of the generalized $R^2$ measures is $1 - \exp(-\chi^{2} / n)$. Even though we still don't know exactly how big $R^2$ must be for the model to be considered to be highly discrimination, $R^2$ is at least unitless.

Answer 3

This seems really really really low. What am I missing here?

Quantities such as the AIC, which involve the use of the log-likelihood, are only meaningful relative to other such quantities. Remember that the likelihood function is defined only up to a scaling constant, so it can be scaled up or down at will. Consequently, the log-likelihood is only defined up to a location constant, and it can be shifted up or down at will. This holds also for the AIC, since this quantity is just the log-likelihood, shifted by a penalty on the number of parameters. That is the reason that it is said that AIC should only be used to compare models.

In computer routines the likelihood function is generally defined directly from the sampling density without removing unnecessary constants, so in this case the scaling issue may not be a factor. In the R Bloggers post you link to, there were $n=800$ data point used in the logistic regression. The log-likelihood from the numbers you give is:

$$\hat{\ell} = (727.9-2 \times 8)/(-2) = -355.95.$$

Thus, the average log-likelihood per data point is $\hat{\ell}/n = -0.4449375$, which corresponds to a likelihood value of $0.6408643$ for a single data point. This is not particularly low, and ought not be cause for any alarm.

Answer 4

You have correctly pointed out that if you back-calculate the likelihood, using the AIC reported by R, you get ridiculously low likelihoods. The reason is that the value of AIC reported by R (call it AICrep) is not the true AIC (AICtrue). AICrep and AICtrue differ by a constant that depends on the measured data but which is independent of the model chosen. Therefore a likelihood back-calculated from AICrep will be incorrect. It is the differences in AICs, when different models are used to fit the same data, that are useful in selecting the best model.