I am currently working with a mixed effects model, and I would like to compare model fit between many potential models. I have seen some posts on this forms (as well as some work in the literature) that discuss how AIC can be used to compare model fit only if the model is estimated through maximum likelihood estimation (not REML), but I am not quite clear on what the consensus is when it comes to model building (see Morrell et al, 2009 or Faraway, 2006).
Posts on this discussion also discuss how with models where the fixed effects are different, AIC cannot be used under REML (see here , REML vs ML stepAIC, Allowed comparisons of mixed effects models (random effects primarily)).
My main questions are as follows:
- How can nested models be compared in mixed models? Will the normal F-tests or Chi-squared suffice?
- Under what conditions can AIC be used to compare models under ML and REML? I gather than AIC can be used under pretty much any circumstance as long as REML is not used. Is this a correct thought?
- Is the following an acceptable model-selection technique?
Use
StepAICor similar to essentially find the fixed effects that produce the lowest AIC (ensuring variables that are included make good theoretical sense), estimating each model with Maximum Likelihood. Then, vary the random effects, comparing AIC while also using ML estimation. Finally, re-estimate the preferred model using REML? - If the above is not valid, what is the consensus as to a similar technique for model building?
I greatly appreciate any help, and would especially appreciate it if you could point me to some recent works in the literature that provide guidance on how to select a mixed model's parameters, particularly covering instances where AIC can and cannot be used.
