Test significance of categorical variable with multiple levels, when model also includes interactions

Question

I am fitting a mixed model with the command

model=lmer(Activity ~ 1 + Novelty*Valence*ROI + (1 | Subject))

Activity is a measure of brain activity, Novelty and Valence are categorical variables coding the type of stimulus used to elicit the response and ROI is a categorical variable coding three regions of the brain that we have sampled this activity from. Subject is an ID number for the individuals the data was sampled from (n=94).

In attempting to get a single estimate for ROI (since it has 3 levels), I ran

model_test=lmer(Activity ~ 1 + Novelty*Valence*ROI - ROI + (1 | Subject))
anova(model,model_test)

The output of my anova command suggests these models are equivalent

object: Activity ~ 1 + Novelty * Valence * ROI + (1 | Subject)
..1: Activity ~ 1 + Novelty * Valence * ROI - ROI + (1 | Subject)
       Df     AIC     BIC logLik deviance Chisq Chi Df Pr(>Chisq)
object 14 -721.88 -641.78 374.94  -749.88                        
..1    14 -721.88 -641.78 374.94  -749.88     0      0          1

Playing around with different models, this seems to happen any time I eliminate an effect or interaction that is qualified by a higher-order interaction. What is the correct way to get an estimate for this categorical variable?

score 3 · Accepted Answer · edited Apr 13 '17 at 12:44

The problem you have encountered is that by removing the main effect for ROI in the model formulation, you have simply reparameterized the model. The main effect is still there, but now it just appears as part of the interaction. For example:

> dt <- read.csv("http://www.bodowinter.com/tutorial/politeness_data.csv")
> 
> lm0 <- lm(frequency ~ attitude*gender, data=dt)
> lm1 <- lm(frequency ~ attitude*gender - gender, data=dt)
> 
>summary(lm0)

Call:
lm(formula = frequency ~ attitude * gender, data = dt)

Coefficients:
                    Estimate Std. Error t value Pr(>|t|)    
(Intercept)          260.686      7.784  33.491   <2e-16 ***
attitudepol          -27.400     11.008  -2.489   0.0149 *  
genderM             -116.195     11.008 -10.556   <2e-16 ***
attitudepol:genderM   15.890     15.664   1.014   0.3135    

Residual standard error: 35.67 on 79 degrees of freedom
Multiple R-squared:  0.7147,    Adjusted R-squared:  0.7038 
F-statistic: 65.95 on 3 and 79 DF,  p-value: < 2.2e-16

> summary(lm1)

Call:
lm(formula = frequency ~ attitude * gender - gender, data = dt)

Coefficients:
                    Estimate Std. Error t value Pr(>|t|)    
(Intercept)          260.686      7.784  33.491  < 2e-16 ***
attitudepol          -27.400     11.008  -2.489   0.0149 *  
attitudeinf:genderM -116.195     11.008 -10.556  < 2e-16 ***
attitudepol:genderM -100.306     11.144  -9.000 9.71e-14 ***

Residual standard error: 35.67 on 79 degrees of freedom
Multiple R-squared:  0.7147,    Adjusted R-squared:  0.7038 
F-statistic: 65.95 on 3 and 79 DF,  p-value: < 2.2e-16

Note that the main effect for gender in lm0 is now shown as the interaction term attitudeinf:genderM in lm1. This is because there are only 2 levels of gender. If we had more than 2 levels of a categorical variable and removed it's main effect, the interaction terms would then be simple contrasts. Note also that the estimate for the interaction parameter in lm0 is now the difference in the two levels of the interaction in lm1

The $R^2$, F statistic and the associated degrees of freedom are identical in both models, because they are simply different parameterizations of the same model.

In order to test the significance of gender we need to remove it from the interaction as well. Since this example is a two-way interaction, we need to remove the interaction entirely:

> lm3 <- lm(frequency ~ attitude, data=dt)
> anova(lm0,lm3)

Analysis of Variance Table

Model 1: frequency ~ attitude * gender
Model 2: frequency ~ attitude
  Res.Df    RSS Df Sum of Sq      F    Pr(>F)    
1     79 100511                                  
2     81 345341 -2   -244830 96.216 < 2.2e-16 ***

Also see this post for more details about testing the significance of a categorical variable.

Test significance of categorical variable with multiple levels, when model also includes interactions

1 Answers1