3

I know that this question sounds familiar to some other, but I believe the responses were not clear in those and were focused on REML models.

I would like to know if it is sensible to compare 2 or more modes that differ in their fixed effects for example

model1 <- lmer(DV ~ Var1 * Var2 + (1 +var2| var4), REML = FALSE)
model2 <- lmer(DV ~ Var5 * Var2 + (1 + var2| var4), REML = FALSE)

So the fixed effects part is different var1 vs var5.

Most answers I find refer to the fact that REML = TRUE model cannot be compared using anova when including different fixed effects, whereas ML models can. But I also find papers and other responses suggesting that using ML (that is REML = FALSE), can only be used to compare different fixed effects if they are nested, which confuses me because I have always thought that what defines the nestedness is the random structure.

Appreciate any pointers/references.

Myriad
  • 119
  • 6
  • 2
    anova performs a likelihood ratio test, which only makes sense if the two compared models are nested. This applies to all uses of LR tests, be it for LMs, LMMs, GLMs, you name it. "Nestedness" has little to do with random effects in this case! Thus, the answer is no: it does not make sense to compare these two models with anova. – Lukas McLengersdorff Sep 08 '20 at 17:16
  • thanks @LukasMcLengersdorff for the response, would it be acceptable to use the AIC in this case for comparison? Or do you have any suggestions on how non nested models can be compared – Myriad Sep 08 '20 at 18:58
  • AIC, BIC are all valid for non-nested models - but if the two models have the exact same number of parameters, comparing AIC/BIC gives you just the same as comparing the likelihoods. I cannot give you any clear recommendations regarding thresholds, though (so I can't tell you what difference in AIC is considered "substantial"). But what exactly are you trying to achieve with this model comparison? Perhaps you can achieve the same goal with a set of nested model comparisons? – Lukas McLengersdorff Sep 08 '20 at 19:51

1 Answers1

5

What you are doing is a kind of stepwise procedure. Please don't select variables with any kind stepwise procedure. See the following for further details on why stepwise is bad:
Algorithms for automatic model selection
What are modern, easily used alternatives to stepwise regression?

Please use knowledge of the variables` relations to each other and the outcome, and ensure that you are properly handling confounding and mediation. A DAG can be very helpful for this. See the following for further details:
How do DAGs help to reduce bias in causal inference?

As mentioned in the comments, anova performs a likelihood ratio test, which is only valid for nested models - that is where the parameters in one model are a subset of the other, which is not the case here.

Also note that in the two models, you include random slopes for var3 but no fixed effect. This is very rarely a good idea, as it assumes that the overall "effect" of the variable is zero.

Robert Long
  • 53,316
  • 10
  • 84
  • 148