2

I have conducted a 2-way ANOVA and obtained a significant Main Effect in IV1 and IV2, but a non-significant interaction. Following that, I have analyzed IV1 using a test for Simple Effects and received a significant Simple effect at level 1 and a non-significant simple effect at level 2.

To speak in particularities of my experiment: I tested the main effect of sex (factor 1) and treatment (factor 2). The interaction between the main effects was non-significant (p = .279), but the main effects were both individually significant (sex, p = .008 and treatment, p = .022). I proceeded to run a simple effects test and identified significance in males (p = .016) and non-significance in females (p = .388). Without the simple effects test I wouldn't be able to distinguish the efficacy of drug treatment in males versus females. Surely the simple effects test was necessary to run in this case, contrary to the non-significant main interaction? Is there something I am missing that would counter my interpretation of the treatment being non-efficacious in females?

I am under the understanding that because of the non-significant interaction, completing a simple effect test was deemed "unnecessary". I don't understand why the simple effect test is only completed following the presence of a significant interaction of main effects. Can anybody explain how the Simple Effect test is influenced by the interaction, and how I would interpret my Simple Effect results accordingly?

Hecealdor
  • 21
  • 3
  • Are you sure you are using the terms "significant" and "non-significant" in the way you intend? Your understanding would be consistent with interpreting these terms in the opposite of their usual meanings. – whuber Jan 14 '21 at 15:50
  • No, I reported the significance of each Effect and Interaction accurately. Are you suggesting that my second paragraph is incorrect? i.e. Simple Effect IS supposed to be tested when the interaction of IV1 and IV2 is non-significant? – Hecealdor Jan 14 '21 at 15:58
  • Normally an insignficant interaction leads you to proceed as if the interaction coefficient is zero. The only things left to test are the simple effects. – whuber Jan 14 '21 at 16:06
  • Interesting. My colleague suggested that running a test for simple effects isn't normally done following the result of no interaction observed. In this case, are my results from the simple effects test valid and acceptable to report? My results being: significance of simple effect at level 1 and non-significance of simple effect at level 2. – Hecealdor Jan 14 '21 at 16:14
  • If you conclude there's no interaction and do not test for the main ("simple") effects, what is left of your model?? – whuber Jan 14 '21 at 16:53
  • It's a trivial analysis, it ends there. I am just baffled as to why the absence of an interaction would influence the decision to NOT test for simple effects. – Hecealdor Jan 14 '21 at 17:47
  • So am I. You ought to check back with your colleague to make sure you understood them correctly and, if so, to ascertain their reasoning. – whuber Jan 14 '21 at 18:40
  • 2
    @whuber: You seem to interpret *simple effect* as *main effect*. I recently learned this is used in another way, see http://glimo.vub.ac.be/downloads/simpleeffect.htm, as a **main effect but restricting to only some level of some other variable**. [Simple slope](http://www.quantpsy.org/interact/mlr2_instructions.pdf) seems to be used in the same sense. – kjetil b halvorsen Jan 15 '21 at 13:21
  • 1
    @kjetil Thank you: that looks right. (I had begun to wonder whether there was some difference of interpretation here, which is why I tried to make mine clear in my previous comment.) Now the question makes sense, if we take the colleague's advice to mean "because you will proceed as if there is no interaction, there is no point in distinguishing simple effects from main effects." – whuber Jan 15 '21 at 14:10
  • I am afraid that I still don't understand the influence of interaction on the simple effects test. My interpretation of the simple effects test is that it reveals the impact of the individual levels of the IV, in a way that the main effects test is incapable of identifying. Precisely how would the absence or presence of an interaction affect the decision to complete the simple effects test? – Hecealdor Jan 15 '21 at 15:00
  • 1
    Think about this: When you split the sample in subgroups and do multiple tests, you get **lower power**. Please also show us your confidence intervals! (and sample sizes.) [This Gelman blog post is relevant](https://statmodeling.stat.columbia.edu/2016/05/25/the-difference-between-significant-and-not-significant-is-not-itself-statistically-significant-education-edition/). Read it! – kjetil b halvorsen Jan 16 '21 at 15:53

1 Answers1

1

Simple effects are defined as the effect of one variable, but restricted only to some level of some other variable. This meaning is used in for instance the R package (on CRAN) phia and discussed in its vignette. Another term which seems to be used with the same meaning is Simple slope.

For a simple illustration, let there be two factors, a and b with interaction a:b. If they have each two levels in a factorial structure, there are two simple effects for a (and likewise for b), one for each level of b. The interaction is the difference between the two simple effects. So, if the interaction is zero, the simple effects are equal (and equal to the main effect.) It is in this sense that there is no need to study simple effects when there is no interaction.

Much the same is true when the factors have more than two levels (or with numerical variables), but the details are more involved. See for instance the vignette linked above.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
  • I'll speak in context of my experiment so it makes more sense.I tested the main effect of sex (factor 1) and treatment (factor 2). The interaction between the main effects was non-significant, but the main effects were both individually significant. I proceeded to run a simple effects test and identified significance in males and non-significance in females. Without the simple effects test I wouldn't be able to distinguish the efficacy of drug treatment in males versus females. Surely the simple effects test was necessary to run in this case, contrary to the non-significant main interaction? – Hecealdor Jan 15 '21 at 21:48
  • 1
    Can you please add this new information as an edit to the Q? Not everybody reads comments. And, some numerical details could be interesting, for a more detailed response. I will try to edit my answer tomorrow, now its night here ... – kjetil b halvorsen Jan 16 '21 at 03:20