0

If you run an ANOVA and find that there are significant interactions, do you then pull the variables out individually and re-run the ANOVA?

I am using Type III. I want to see if there is a difference in survival between 3 groups, but I have to include 3 variables other than group in my model to make sure they didn't influence survival as well. When I do that A affects survival, and so do A*D and A*E, but when I run A, D, and E by themselves, nothing affects survival.

I wish had a better grasp on stats, because I am still so confused. In essence, I am looking at how survival differed between 3 groups of animals while taking into account that length, weight, and age may have played a role. When I look at these factors individually, none had a significant effect on survival...but, when I include interaction terms (i.e. length*weight, group*weight, etc.) the variables are significant. What does this mean? Am I wrong to put in my results that x,y, and z do not affect my results?

Michele
  • 1
  • 1
  • Why do you mean by “pull the variables out individually”? Run an ANOVA with an interaction but no main effects? Run separate ANOVA with each factor in turn? – Gala Jun 15 '13 at 23:04
  • Example: I ran a full model. A was sig, A*D was sig, A*E was sig, but B was not and C was not nor were any of the interaction terms that included B and C. Do I rerun the ANOVA with A, D, and E individually to see which variable actually influenced my results? – Michele Jun 15 '13 at 23:07
  • 1
    You could conceivably leave B and C out but that should only affect things in specific instances. What kinds of sums of squares are are you using for your ANOVA? Do you know if there is any correlation among your predictors? What, specifically do you really want to find out. (It's better if you edit your question, not just add a comment.) – John Jun 15 '13 at 23:26
  • I wish had a better grasp on stats, because I am still so confused. In essence, I am looking at how survival differed between 3 groups of animals while taking into account that length, weight, and age may have played a role. When I look at these factors individually, none had a significant effect on survival...but, when I include interaction terms (i.e. length*weight, group*weight, etc.) the variables are significant. What does this mean? Am I wrong to put in my results that x,y, and z do not affect my results? – Michele Jun 16 '13 at 01:15

1 Answers1

3

This is akin to a stepwise procedure and generally not recommended. Adjusting a model is good for prediction or for explorative purposes but it biases p-values if those are computed afterwards on the same dataset. Ideally, if the main objective is testing then you should specify the model in advance and stick to that.

You also need to take into account the fact that the difference between significant and not-significant is not itself significant, there is really nothing special about conventional thresholds. This means that if the p-value is .04 for one factor and .06 for another, you have very little evidence that the first is somehow more important than the second one. Formally, you cannot reject the null hypothesis in the second case but that is not evidence than it is true either. Removing one but not the other based only on this seems arbitrary.

Furthermore, what purpose would it serve? If you don't have too many potential variables or particular issues with your model, there is no reason not to include all your covariates/predictors.

See also Frank Harrell's work and previous questions on this site:

Community
  • 1
Gala
  • 8,323
  • 2
  • 28
  • 42
  • I wish had a better grasp on stats, because I am still so confused. In essence, I am looking at how survival differed between 3 groups of animals while taking into account that length, weight, and age may have played a role. When I look at these factors individually, none had a significant effect on survival...but, when I include interaction terms (i.e. length*weight, group*weight, etc.) the variables are significant. What does this mean? Am I wrong to put in my results that x,y, and z do not affect my results? – Michele Jun 16 '13 at 01:15
  • +1, lots of good advice here. However, I wonder if the OP's main issue is how to understand the nature of interactions. @user26934, I wrote an answer here: [interaction-in-generalized-linear-model](http://stats.stackexchange.com/questions/49002//49701#49701), that tries to explain interactions; you may find it helpful. – gung - Reinstate Monica Jun 16 '13 at 01:25
  • I understand the definition of interactions, but not the usage of it in SAS. You do things one way and you get significance. You turn around and do it the other way, and there is no significance. This dilemma has lead me into a world of confusion and, despite all that I have read thus far, I am utterly confused. – Michele Jun 16 '13 at 01:44