ANOVA - Adding interaction term changes a main effect to being insignificant. How can that be?

Question

UPDATE: I don't think the single posted answer is correct. When I run summary(lm(DependentVar ~ IndVar1 * IndVar2, data = Data1)) the first listing is IndVar11 @ p = 0.113, and the four IndVar1:IndVar2 entries are all non significant (p = 0.16 to p = 0.41).

Regarding the suggested duplicate - the accepted answer there is about small p-value differences and small samples sizes, neither of which is the case here. In that answer, the author says he doesn't see other way this can happen. So what is happening here?

I'm looking for an explanation as to how an independent variable can go from being highly significant to being highly insignificant when an interaction term is added. IndVar1, below.

IndVar1 has 2 levels. IndVar2 has 5 levels.

Thanks

> Model_1 <- aov(DependentVar ~ IndVar1 + IndVar2, data = Data1)
> Anova(Model_1, type="III")
Anova Table (Type III tests)

Response: DependentVar
             Sum Sq   Df    F value    Pr(>F)    
(Intercept) 18486.4    1 11622.9984 < 2.2e-16 ***
IndVar1         23.7    1    14.8532 0.0001353 ***
IndVar2    39.7    4     6.3382 5.711e-05 ***
Residuals    2175.1 1341                         
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  
  
> Model_1 <- aov(DependentVar ~ IndVar1 * IndVar2, data = Data1)
> Anova(Model_1, type="III")  
Anova Table (Type III tests)

Response: DependentVar
                    Sum Sq   Df   F value    Pr(>F)    
(Intercept)        15473.4    1 9781.1865 < 2.2e-16 ***
IndVar1                 4.3    1    2.7758 0.1131780    
IndVar2           36.8    4    5.8247 0.0001231 ***
IndVar1:IndVar2    14.1    4    2.2124 0.0666647 .  
Residuals           2149.1 1337                        
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
>

This is a FAQ: see https://stats.stackexchange.com/search?tab=votes&q=interaction%20signif*%20change. — whuber, Sep 20 '20 at 17:30
The accepted answer at "The question already has an answer here" is about small differences in significance, and small samples sizes (using up degrees of freedom). Neither of which is the case here. There is a very large sample, and very large p-value differences. The author of the accepted answer specifically says those are the only cases where he thinks this is possible. So what's happening here? — Kyle, Sep 20 '20 at 17:39
Have you looked at more of the hits in the link? Variations of your question abound on this site, concerning all conceivable permutations of adding or removing variables that increase or decrease significance levels in all kinds of regression models. I'm sure you can find situations essentially the same as yours as well as good advice. One aspect that rarely matters in such circumstances is the amount of data. As far as interactions go, you can find loads of answers pointing out that they usually are almost collinear with their component variables. — whuber, Sep 20 '20 at 17:44
I don't think you can center factors, so I don't find any help there. — Kyle, Sep 20 '20 at 18:44
Of course you can center them! Factors must be represented as vectors in the model matrix and, as such they can be centered. Indeed, that's exactly what happens in most algorithms. — whuber, Sep 20 '20 at 21:19
Somewhat related to this is the possible issue of multicollinearity. Is it possible that your two factors are correlated? Without knowing more about your data and research methods, it's hard to know what might be going on. Seems like it may be something to do with your data, so checking for influential cases, plotting the variables, and doing some more explorations may help you understand what's causing the issue (assuming it's not a collinearity issue). — Billy, Sep 23 '20 at 14:25
I think it's problematic that a question is closed to new answers, but is otherwise open. — Kyle, Sep 23 '20 at 16:58
I suppose I don't know how a correlation could be done for two factors. They can conceptually be correlated (e.g., non-independent in observational/quasi-experimental designs), but unless there is some kind of order implied in the levels of the factors, it doesn't make sense to compute something like a Pearson correlation. If there's some inherent order to the levels, then you could use ordered() instead of factor() for the variable. Then this becomes a regression instead of a traditional ANOVA (though note that these models are just specific cases of the general linear model) — Billy, Sep 23 '20 at 20:46
@Billy -Sorry, the interaction between them on the dependent variable, without the covariate in the model, has p = 0.73. — Kyle, Sep 23 '20 at 22:25

score 5 · Accepted Answer · answered Sep 20 '20 at 04:06

In a model with interactions, the interpretation of the IndVar1 term is not 'the effect' of IndVar1, or even 'the association', since the whole point of the interaction is that the association of IndVar1 with the outcome is different for different value of IndVar2.

So, what is it? Well, for Type III sums of squares, it's the estimated association at the value of IndVar2 coded as zero. In this case, if you haven't changed any of the defaults, that will be the estimated association between IndVar1 and the outcome at the reference category (first category) of IndVar2. There's very waek evidence for the association being non-zero

This could happen for multiple reasons, which we can't distinguish because you haven't told us enough

maybe the association is actually weak at that level of IndVar2
maybe there are very few observations at that level of IndVar2, so you can't tell if the association is actually weak or strong
maybe the observations at that level of IndVar2 mostly have the same value of IndVar1, so again you can't tell if the association is actually weak or strong

As an example of the first type, might think of income by age-group and gender, where you could see a strong effect of age, a strong effect of gender, and an interaction -- but if your lowest age group was 0-10, there wouldn't be much gender difference in that age group because none of them are earning much.

Regarding defaults, I ran this to correct Type III sums from car::Anova() : options(contrasts = c("contr.sum", "contr.poly")) — Kyle, Sep 20 '20 at 09:17
I don't think this answer is correct. When I run summary(lm(DependentVar ~ IndVar1 * IndVar2, data = Data1)) the first listing is IndVar11 @ 0.113, and the four IndVar1:IndVar2 entries are all non significant (p = 0.16 to p = 0.41). — Kyle, Sep 20 '20 at 09:29
This STUPID system won't let me remove my upvote. I'm not too happy about that. — Kyle, Sep 20 '20 at 17:09
@Kyle There are good reasons for that. You will be able to alter your vote if and when this post is edited. Moderators have no power to change or remove votes -- we can't even find out who voted! — whuber, Sep 20 '20 at 17:28

ANOVA - Adding interaction term changes a main effect to being insignificant. How can that be?

1 Answers1