4

I am running a mixed factors ANOVA for a brain imaging study on language processing. The design includes four within-subject factors:

  1. complexity: simple/complex;
  2. agreement: correct/number violation/gender violation;
  3. hemisphere: left/right;
  4. region: anterior/posterior;

and one between-subject factor:

  1. proficiency: low/high proficiency.

Factors 3 and 4 (hemisphere, region) only matter if they interact with the other factors. This is because the effects of complexity and agreement are predicted to emerge in specific areas of the scalp.

The results reveal a significant 5-way interaction (complexity by agreement by hemisphere by region by group), and I find it almost impossible to figure out what is driving the interaction by just looking at the means.

Am I justified in looking at the two proficiency groups separately? Most similar studies do this, but I'm unclear as to how to do this. For example, once I decide to examine proficiency learners separately, I find a 3-way interaction between agreement, hemisphere, and region in the high-proficiency learners (meaning that there is a brain response for agreement violations, which is captured in the left anterior portion of the scalp), but there is no such interaction for low-proficiency learners (p = .13) Is it licit to report this difference even if the original omnibus ANOVA (with group as a factor) showed that there was no agreement by hemisphere by region by proficiency interaction?

In other words, once I decide to look at the two proficiency groups, can I run the analysis as if I had never compared the two groups or am I constrained by the original ANOVA as to what follow-ups I can do?

Gala
  • 8,323
  • 2
  • 28
  • 42
José Alemán Bañón
  • 41
  • 2
  • 3
    You can Google general advice on complex interactions (but it probably won't be of much use). It would be best if you could post or link to the table of condition means. Also, report the magnitude and size of the lower order interactions and main effects. My first approach would be to graph it. You can probably do it in 4 panel graphs with 3 levels within each graph. – John Jul 19 '13 at 00:55

1 Answers1

8

Without the numbers I can just give the general advice, graph it. You have 2 x 3 x 2 x 2 x 2 so that's not that hard. Take one of your 2 level variables and calculate it's effects across everything else. That's what you'll plot. You can now plot 3x2 or 2x3 panel graphs. You'd have 4 panels with the remaining two factors crossing them, one divided horizontally, and the other vertically.

Stare at that for awhile.

Maybe several permutations. Change what's on the x-axis, the panels, line types, and effects through as many iterations as you need until you come to an understanding. If you've got some theoretical reasons for directions of causality take the highest level and make it panels and the lowest and make it your line types or x-axis.

Hopefully you'll come up with something. This could take a lot of work and a long time to see the nature of the relationship. But going through several versions of the graphs can often allow you to see what's going on.

(I recently had to do the same thing for 10 x 3 x 2 x 2 x 2. It took months looking at graphs off and on and letting them incubate before the nature of the 5 way came through.)

UPDATE:I just noticed your question on splitting by proficiency. What you described is an absolute no-no. An interaction in one proficiency group while none in another does not tell you that there are differences between those groups. An interaction with proficiency would. So leaving proficiency out of your interactions gives you much less information. Your interactions mean something and the meaning is in the data, not in more tests.

As I ask most of my students, if you're going to test everything afterwards what was the point of doing an ANOVA in the first place?

John
  • 21,167
  • 9
  • 48
  • 84
  • Thanks a lot, John. This helped a lot. In this type of research, you usually plot the brain responses for the conditions under investigation (in this case: grammatical, number violation, gender violation) on a scalp map that represents the two brain hemispheres and the anterior/posterior portion of the head. So that kind of map takes care of two of the factors (hemisphere, anteriority) and functions as an interaction plot. I did this for the two proficiency groups to figure out what was going on, and it helped me decide that follow-ups to conduct. – José Alemán Bañón Jul 23 '13 at 16:03
  • I also looked at all of the interaction plots that SPSS spits out and, of course, it was consistent. I felt a little uncomfortable deciding what follow-ups to do based on the visual observation of the means, but then again, I do not call those comparisons "planned comparisons" and I still corrected for Type I error, so it should be okay. Thanks! – José Alemán Bañón Jul 23 '13 at 16:04
  • 1
    Just make sure that you extract as much information out of the interaction itself as you can and don't make inferential errors on followups. When all of your levels are 2 there is almost never any need for followup tests because the interactions tell you that what you see is significant. And NEVER do follow up tests and use something like x1:x2 was significant while y1:y2 was not therefore...The interaction already told you they were different while a followup like that does not. – John Jul 23 '13 at 16:13
  • Hi again, John. The main variable of interest in my study is agreement, which has 3 levels (grammatical, number violation, gender violation). So, I am running post-hoc tests, because I need to know whether number differs from grammatical, whether gender differs from grammatical, and whether number differs from gender. Does that sound reasonable? Thanks! – José Alemán Bañón Jul 24 '13 at 16:34
  • It's unlikely you need the post hoc tests. The tests will probably make things worse inferentially. What if the scores are 3, 4, 5? Minimally the 3 and 5 will be different so you know that already but it's possible neither will be different from 4. What do you conclude from all of your tests? The ANOVA conclusion is there is a main effect and you describe the data, done. I have yet to see a 3 level ANOVA main effect where one's inference is improved by post hoc tests. For 5 and greater they can certainly be handy though. – John Jul 24 '13 at 16:47
  • But then how do you know what's driving the interaction? Just by looking at the graphs? The example you provided is very similar to what happens in another longitudinal study where we find an interaction between agreement and testing session. By looking at the graphs, learners did not treat grammatical sentences and violations differently at the first testing (they were novice learners, so this is expected), but they did at the second and third testing sessions. However post-hocs only show the effect for the second session. The third session does not differ from either the 1st or 2nd. – José Alemán Bañón Jul 24 '13 at 17:32
  • This can go on and on...post an interaction you think isn't explained purely by the pattern and interaction and asked if tests are needed. Also post the statistics. In that new question you could get some clarification. I'm not saying it's impossible to need post-hocs, just unlikely. (In your stated example I don't trust your post-hocs because you've got a Type I, Type II error trade off. If your second and third sessions looked similar and different from first I'd go with that. Post-hocs aren't some ultimate arbiter of truth.) – John Jul 24 '13 at 18:07