7

I am in the process of calculating the akaike information criterion (AIC) for a set of 15 nested models. Data was generated from the 5th model and used in a parameter estimation for all models. This data is time course data from a set of ODE's consisting of 4 variables over two different initial conditions. Therefore $2*4*31=248$ observations. My task is to infer that the data was generated from the 5th model using AIC. The AIC form I am using is:

$$-N*ln[(RSS/N)]+2k+[(2k(k+1))/(N-k-1)]$$ where:

$N$: number of observations (248 in my case)

$k$: number of estimated parameters plus 1 (since the error is a parameter apparently).

Here are my calculations are in the table below (copied from Matlab). As you can see the estimation identified the 5th model in the series as only the 2nd best option (by way of minimizing the sum of squares). I then proceeded to calculate each of the AIC terms individually then sum at the end.

What I observe is that the model that was ranked 1st by chi squared was ranked last in AIC. Further, there appears to be an inverse relationship between chi squared and AIC in that the second ranked best model by chi squared is the second worst by AIC, and so on.

My question is: has a relationship of this nature been observed before? If so where and if not (which I suspect to be the case) what am I doing wrong?

chisq      chisq_rank num_parameters  1st_term  2nd_term   3rd_term    AIC (sum)
1.60175856506195    9   8   1250.49700109714    16  0.602510460251046   1267.09951155739    7
2.22401333127733    15  8   1169.10061312778    16  0.602510460251046   1185.70312358803    1
2.21538124402173    13  10  1170.06505109382    20  0.928270042194093   1190.99332113601    3
1.50205042169781    8   9   1266.43621067286    18  0.756302521008403   1285.19251319387    8
0.593373856737609   2   9   1496.76912499725    18  0.756302521008403   1515.52542751826    14
1.26848304390402    4   10  1308.35053928393    20  0.928270042194093   1329.27880932612    12
0.593367670012715   1   10  1496.77171074611    20  0.928270042194093   1517.69998078830    15
2.22098980957599    14  8   1169.43799579043    16  0.602510460251046   1186.04050625068    2
1.45989299143802    6   8   1273.49627045328    16  0.602510460251046   1290.09878091353    10
2.01998433198069    10  9   1192.96406982433    18  0.756302521008403   1211.72037234534    6
2.02269351982039    11  9   1192.63167693553    18  0.756302521008403   1211.38797945654    5
1.04998188601805    3   8   1355.23464672182    16  0.602510460251046   1371.83715718207    13
2.03684592732117    12  9   1190.90250971104    18  0.756302521008403   1209.65881223205    4
1.50205035842807    7   9   1266.43622111918    18  0.756302521008403   1285.19252364019    9
1.30853041112627    5   9   1300.64196743421    18  0.756302521008403   1319.39826995522    11
CiaranWelsh
  • 153
  • 9
  • What does $\chi^2$ stand for? What does it indicate? – Richard Hardy Nov 16 '15 at 18:48
  • Chi squared is the sum of squares - the minimized distance between the experimental data (but in this case generated from one of the models) the corresponding model simulated data – CiaranWelsh Nov 16 '15 at 20:12
  • So it is residual sum of squares? If so, it would be proportional to $1-R^2$ (I find it convenient to think in terms of $R^2$ than $\chi^2$). – Richard Hardy Nov 16 '15 at 20:16
  • Yes, the residual sum of squares. Are you using $R^2$ synonymously with $chi squared$ here just different notation or is it something else? (I'm a biologist background) – CiaranWelsh Nov 16 '15 at 20:23
  • Oh, I thought $R^2$ will be known to you. It is the [coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination). And no, it is not a synonym of $\chi^2$. – Richard Hardy Nov 16 '15 at 20:26
  • I thought it could be Pearsons correlation coefficient but couldn't see what that would have to do with Chi Squared. Thanks for responding - gives me something new to go on. – CiaranWelsh Nov 16 '15 at 20:29

1 Answers1

1

Check your formula for AIC against the Wikipedia AIC page, equal-variances case. The negative sign before your first term seems to be in error. (That section of the Wikipedia page omits the correction from AIC to AICc, the third term of your equation, which is discussed higher up on that page.)

There might be some confusion because the calculated AIC values will then appear to be negative, which would seem to be wrong. That's because the formula that you intended to use omits a constant, as explained on the Wikipedia page. For model comparisons on the same data set the constant can be ignored and you still base your selection on the lowest calculated AIC.

Also, note that yours aren't nested models in the usual sense, as many have the same number of parameters. Usually the "nested" terminology refers to a set of models with one large set of predictor variables in the first model, the second model containing only a proper subset of those predictors, the third containing a subset of the predictors in the second model, and so forth in strictly decreasing numbers of predictors. So there cannot be 2 of a set of nested models having the same number of predictors. There is some dispute over the validity of AIC for non-nested models, as noted on this Cross-Validated page.

Community
  • 1
EdM
  • 57,766
  • 7
  • 66
  • 187
  • Hi EdM, thanks for responding. Some of the models may have the same number of parameters but they are not all the same parameters. The models here basically all contain the same core network of ODEs but have different additional hypotheses in attempt to explain the data (in accordance with the principles of systems biology). However, if you are correct and my models are not nested, is there a criterion that you would recommend? Is BIC any better? – CiaranWelsh Nov 17 '15 at 09:24
  • 1
    @user3059024 AIC is still a useful choice for non-nested models and is often used in practice. I raise the caution so you will know that the theoretical underpinnings of AUC might not be so strong for non-nested models as they are for strictly nested models. See [this page](http://stats.stackexchange.com/q/577/28500) for discussion of AIC versus BIC. In practice, BIC places a higher penalty on model complexity. Some would suggest to examine both. In your case the log-likelihood (first) term is much larger in magnitude than the penalty term, so ordering by AIC and BIC might be identical. – EdM Nov 17 '15 at 14:25