15

I am currently reviewing some work and have come across the following, which seems wrong to me. Two mixed models are fitted (in R) using lmer. The models are non-nested and are compared by likelihood-ratio tests. In short, here is a reproducible example of what I have:

set.seed(105)
Resp = rnorm(100)
A = factor(rep(1:5,each=20))
B = factor(rep(1:2,times=50))
C = rep(1:4, times=25)
m1 = lmer(Resp ~ A + (1|C), REML = TRUE)
m2 = lmer(Resp ~ B + (1|C), REML = TRUE)
anova(m1,m2)

As far as I can see, lmer is used to compute the log-likelihood and the anova statement tests the difference between the models using a chi-square with the usual degrees of freedom. This does not seem correct to me. If it is correct, does anyone know of any reference justifying this? I am aware of methods relying on simulations (Paper by Lewis et al., 2011) and the approach developed by Vuong (1989) but I do not think that this is what is produced here. I do not think that the use of the anova statement is correct.

amoeba
  • 93,463
  • 28
  • 275
  • 317

1 Answers1

8

This is not correct in two ways:

  1. (Ordinary) likelihood ratio test can only be used to compare nested models;
  2. We cannot compare mean models under REML. (This is not the case here, see @KarlOveHufthammer's comments below.)

In the case of using ML, I am aware of using AIC or BIC to compare the non-nested models.

Randel
  • 6,199
  • 4
  • 39
  • 65
  • 9
    Regarding point 2, the `anova()` function in R does *not* compare the two models fitted under REML; it refits them using ML and then perform the test. See `lme4:::anova.merMod`, which contains the line `mods – Karl Ove Hufthammer Jan 29 '14 at 19:32
  • 2
    also note that there is some disagreement on nesting: [Brian Ripley says nesting is essential for AIC comparison](http://www.stats.ox.ac.uk/~ripley/ModelChoice.pdf‎) (see p. 20 of linked document for discussion), while [Anderson and Burnham](http://warnercnr.colostate.edu/~anderson/PDF_files/AIC%20Myths%20and%20Misunderstandings.pdf) (see p. 2) disagree .. – Ben Bolker Jan 29 '14 at 23:43
  • 1
    It would be nice to try to implement a test of whether the models passed to `anova.merMod()` were nested or not, but it could be hard to do completely reliably/generally ... – Ben Bolker Jan 29 '14 at 23:45
  • 2
    @BenBolker [Another reference](http://warnercnr.colostate.edu/~anderson/PDF_files/AIC%20Myths%20and%20Misunderstandings.pdf) (see also [this](http://en.wikipedia.org/wiki/Akaike_information_criterion#How_to_apply_AIC_in_practice) and [this](http://stats.stackexchange.com/q/20441/37729)) for the use of AIC with non nested models, as long as you consider all the normalising constants as well as non-pathological models. In the context of LMM, however, you have to use some [modifications](http://biomet.oxfordjournals.org/content/97/4/773.abstract) of the AIC. – LessFaceMoreBook Jan 29 '14 at 23:52
  • @LessFaceMoreBook: your first link ("another reference") is the same as mine. Ripley would disagree with all of the above links ... http://glmm.wikidot.com/faq#aic has some discussion of AIC/LMM issues. – Ben Bolker Jan 30 '14 at 00:00
  • @BenBolker, thanks for bringing this up! It's good to know more ideas. But the link for Prof. Ripley's reference seems not work, `404 Not Found`. – Randel Jan 30 '14 at 19:36
  • 2
    Link mangled: I think http://www.stats.ox.ac.uk/~ripley/ModelChoice.pdf should work. – Ben Bolker Jan 30 '14 at 20:38
  • 3
    @BenBolker Well, Brian Ripley is quite opinionated. However, he hasn't provided a devastating argument against the use of AIC for non nested models :). Sorry for repeating your link. – LessFaceMoreBook Jan 30 '14 at 23:46