why different results with summary()and anova() function in linear mixed effect model

Question

I know that the question has been submitted several times but given my level of knowledge, I'm afraid I haven't been able to find a satisfactory answer. I have two fixed effects : frequency (high vs low) and predictability (high vs low). my model is as follows :

m1=lmer(log(FF) ~ frequence * Predic + (1 | Sujet) + (1 | item), data= FreqPredicExpert72)

The anova function returns : 

Type III Analysis of Variance Table with Satterthwaite's method
                  Sum Sq Mean Sq NumDF  DenDF F value    Pr(>F)    
Predic           1.78800 1.78800     1 68.035 23.1208 8.774e-06 ***
frequence        0.78005 0.78005     1 68.155 10.0869  0.002243 ** 
Predic:frequence 0.16368 0.16368     1 67.894  2.1166  0.150321   

The summary function returns : 


Linear mixed model fit by REML. t-tests use Satterthwaite's method ['lmerModLmerTest']
Formula: log(FF) ~ Predic * frequence + (1 | Sujet) + (1 | item)
   Data: FreqPredicExpert72
REML criterion at convergence: 558.9
Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-3.6928 -0.5517 -0.0021  0.5770  4.3347 
Random effects:
 Groups   Name        Variance Std.Dev.
 item     (Intercept) 0.002058 0.04536 
 Sujet    (Intercept) 0.014550 0.12062 
 Residual             0.077333 0.27809 
Number of obs: 1580, groups:  item, 72; Sujet, 28
Fixed effects:
                     Estimate Std. Error       df t value Pr(>|t|)    
(Intercept)           5.33733    0.02883 49.82410 185.118   <2e-16 ***
PredicHP             -0.05921    0.02490 67.27315  -2.378   0.0202 *  
frequenceHF          -0.03042    0.02489 67.28622  -1.222   0.2260    
PredicHP:frequenceHF -0.05133    0.03528 67.89410  -1.455   0.1503    

Correlation of Fixed Effects:
            (Intr) PrdcHP frqnHF
PredicHP    -0.428              
frequenceHF -0.429  0.497       
PrdcHP:frHF  0.302 -0.705 -0.705

There is no difference concerning p value fonte interaction, but as on can see, anova returns a significant effect of frequency (p = 0.002243 ** ) while this effect is not significant in results with summary (p = 0.2260). For me it's a big problem.

My question is threefold

1) Why both p value of the interaction and p value of the Predic effect are identical in summary() and in anova() function? And why it is not the case for the frequency effect?

2) In what outcome can we trust? (Anova or summary?). Pratically, effect of frequency amounted to 6 ms in the group of 21 adults. It seems unlikely that this effect is significant. So I will tend to rely on results from the summary() rather than anova()

3) What I have just described is part of my data. IN fact, I have three groups of participants (grade 3, grade 5 and Adults).

To analyse my data, I conducted a 3 way-anova : Group x frequency x Predic The model used is as follows :

m1=lmer(log(FF) ~ Group * frequence * Predic + (1 | Sujet) + (1 | item), data= FreqPredicExpert72)

For this model, anova() and summary() returns very different results. If I understand correctly, it’s because summary() function tests contrasts? Is it possible to specify something so that summary function gives global effect for the Group x frequency x Predic interaction as in anova() function. Does this have anything to do with contrast specification?

Answer 1

This apparent discrepancy comes from the difficulty in interpreting "main effects" (e.g., the coefficient for frequence reported by summary(), or a Type III Anova test) when there is an interaction.

With the standard treatment coding default in R, the significance test for frequence reported by summary() in your interaction model will examine whether the coefficient is different from 0 when predictability is at its reference level. It looks like the reference levels for frequence and predictability are both set at "low" in your model. As a concrete example say that there is a significant interaction between the two predictors, with 0 influence of frequence on outcome when predictability is "low" but a large influence when predictability is "high".

Then the coefficient reported by summary() for frequence should be insignificant, with a coefficient of 0! That's because at the reference level of predictability there is no influence of frequence on outcome. That doesn't mean that frequence is unimportant. You would still have a large interaction coefficient involving it. And if you simply set the reference level for predictability to "high" then the coefficient reported by summary() for frequence would now be significantly different from 0.

So you have to be very circumspect when interpreting apparent "main effects" when predictors are included in interactions. As another example, just centering a continuous predictor can change the apparent "significance" of individual coefficients for all predictors interacting with it.

In your case there is an interaction coefficient (-0.05133) of magnitude similar to that of the "main effect" of predictability (-0.05921, for the case of "low" frequence), even though it didn't reach the magical threshold of 5% significance. The combination of the interaction term and the "main effect" of frequence (-0.03042) would estimate an effect of frequence at "high" predictability of -0.08175, which might well pass that threshold. Try switching the reference level of predictability and see for yourself.

This difficulty with interpreting the results of summary() will only be compounded in your final 3-way interaction model. The "main effects" for each predictor would be when both of the others are at their reference levels, however those were chosen.

Type III Anova can lead to similar difficulties in interpretation, as the significance of frequence will be assessed after the contributions of both predictability and the interaction between frequence and predictability are evaluated. So an "insignificant" effect of frequence doesn't mean it isn't important, just that its "main effect" is dwarfed in part by its interaction with predictability. As @gung puts it in this answer, there is an inherent problem with Type III Anova that can lead to false negative results:

If you sum the type III [sum of squares] SS in an ANOVA table, you will notice that they do not equal the total SS. In other words, this analysis must be wrong, but errs in a kind of epistemically conservative way.

Or, as the Warning in the manual page for the car Anova() function puts it:

Be careful of type-III tests.

In your particular case, one might argue that the lack of a statistically significant interaction term (at least in the 2-predictor model) could allow you to revert to a simple additive model. Then this problem would go away, although you should report that you first tested the interaction model.

What you presumably care most about is whether frequence is associated with outcome whether or not that association is primarily due to an interaction with predictability. With fixed-effect models the rms package in R handles this nicely by doing a Wald-type test of whether all of the coefficients (up to 2-way interactions) involving a predictor are 0, taking into account the covariances among the coefficient estimates. I don't work much with mixed models but there should be ways to do the same analysis for those. That might be how Anova() in car performs its Wald tests (as opposed to its default likelihood-ratio tests), but I'm not sure. You might also consider the emmeans package as a tool for evaluating effects of interest in mixed models with interaction terms.

What I'm still confused about is why the "Type III" test reported by anova() from the lmerTest package gave a significant coefficient for frequence while the other Type III tests didn't. But that's seems to be more a question of software implementation than a statistical issue.