Model-selection for linear mixed models over alternative sets of parameters (nlme function in R)

Question

My models look like:

lme1 = lme(y~X+Y+V, random=~1|Subject, data=mydata, method ="ML")
lme2 = lme(y~X+Y+V2+V3, random=~1|Subject, data=mydata, method ="ML")
lme3 = lme(y~X+Y+V4, random=~1|Subject, data=mydata, method ="ML")

where X and Y are factors, but V, V2, V3,and V4 are continuous variables (modeled as covariates). I am using Method ="ML" in the hope that I could compare the likelihood values across the models.

My research question has to do with whether V4 (in lme3) was a better predictor than V2 and V3 together, V2+V3 was better than V, etc. What goodness of fits measure is valid here? Can I use AIC values to compare models of different sets of parameters?

I've also found some references on computing $R^2$ for mixed models. In particular, I am interested in the likelihood ratio test $R^2$ (Magee, 1990) which computes a $R^2$ by comparing each of these models to the null model. Using this method, I'd be comparing all three of my models to the same null model with just y~1. Is it then a valid approach to compare the $R^2$s generated?

I am not a statistician but I would like to use a valid (at least justifiable) measure for my analysis. Any feedback would be greatly appreciated.

Answer 1

I would use Akaike’s Information Criterion ($AIC$) for model selection where: $$ AIC = -2ln(L)-2k $$ Though a better alternative is often $AIC_c$, which is the second-order Akaike’s Information Criterion ($AIC_c$). $AIC_c$ is corrected for small sample size with an addtion bias-correction term because $AIC$ can perform poorly when the ratio of sample size to the number parameters in the model is small (Burnham and Anderson 2002). $$ AIC_c = -2ln(L)-2k+\frac{2k(k+1)}{(n-k-1)} $$ In fact, I would always use $AIC_c$ since the bias-correction term goes to zero as sample size increases. However, there are some types of models where it is difficult to determine sample size (i.e., hierarchical models of abundance see links to these model types here).

$AIC$ or $AIC_c$ can be recaled to $\mathsf{\Delta}_i=AIC_i-minAIC$ where the best model will have $\mathsf{\Delta}_i=0$. Further, these values can be used to estimate relative strength of evidence ($w_i$) for the alternative models where: $$ w_i = \frac{e^{(-0.5\mathsf{\Delta}_i)}}{\sum_{r=1}^Re^{(-0.5\mathsf{\Delta}_i)}} $$ This is often refered to as the "weight of evidence" for model $i$ from the model set. As $\mathsf{\Delta}_i$ increases, $w_i$ decreases suggesting model $i$ is less plausible. Also, the weights of evidence for the models in a model set can be use in model averaging and multi-model inference.