Post hoc test in a 2x3 mixed design ANOVA using SPSS?

Question

I have two groups of 10 participants who were assessed three times during an experiment. To test for differences between groups and across the three assessments, I ran a 2x3 mixed design ANOVA with group (control, experimental), time (first, second, three), and group x time. Both time and group resulted significant, besides there was a significant interaction group x time.

I don't know very well how to proceed to further check for the differences between the three times of assessments, also respect to group membership. In fact, at the beginning I only specified in the options of the ANOVA to compare all the main effects, using the Bonferroni's correction. However, then I realized that this way they were compared the differences in time of the total sample, without group distinction, am I right?

Therefore, I searched a lot on the internet to find a possible solution, but with scarce results. I only found 2 cases similar to mine, but their solutions are opposite!

1. In an article, after the mixed design, the authors ran 2 repeated measures ANOVA as a post-hoc, one for each group of subjects. This way, the two groups are analysed separately without any correction, am I right?
2. In a guide on the internet, they say to add manually in the SPSS syntax COMPARE(time) ADJ(BONFERRONI), just after /EMMEANS=TABLES(newgroup*time), while running the mixed ANOVA. This way, the three times are compared separately for each group, with Bonferroni correction, am I right?

What do you think? Which would be the correct way to proceed?

Answer 1

Answer edited to implement encouraging and constructive comment by @Ferdi

I would like to:

1. provide an answer with a full contained script
2. mention one can also test more general custom contrasts using the /TEST command
3. argue this is necessary in some cases (ie the EMMEANS COMPARE combination is not enough)

I assume to have a database with columns: depV, Group, F1, F2. I implement a 2x2x2 mixed design ANOVA where depV is the dependent variable, F1 and F2 are within subject factors and Group is a between subject factor. I further assume the F test has revealed that the interaction Group*F2 is significant. I therefore need to use post hoc t-tests to understand what drives the interaction.

MIXED depV BY Group F1 F2 
  /FIXED=Group F1 F2 Group*F1 Group*F2 F1*F2 Group*F1*F2 |  SSTYPE(3) 
  /METHOD=REML 
  /RANDOM=INTERCEPT | SUBJECT(Subject) COVTYPE(VC) 
  /EMMEANS=TABLES(Group*F2) COMPARE(Group) ADJ(Bonferroni)
  /TEST(0) = 'depV(F2=1)-depV(F2=0) differs between groups' 
    Group*F2 1/4 -1/4 -1/4 1/4 
    Group*F1*F2 1/8 -1/8 1/8 -1/8 -1/8 1/8 -1/8 1/8 
  /TEST(0) = 'depV(Group1, F2=1)-depV(Group2, F2=1)' Group 1 -1
    Group*F1 1/2 1/2 -1/2 -1/2 
    Group*F2 1 0 -1 0  
    Group*F1*F2 1/2 0 1/2 0 -1/2 0 -1/2 0 .

In particular the second t-test corresponds to the one performed by the EMMEANS command. The EMMEANS comparison could reveal for example that depV was bigger in Group 1 on the condition F2=1.

However the interaction could also be driven by something else, which is verified by the first test: the difference depV(F2=1)-depV(F2=0) differs between groups, and this is a contrast you cannot verify with the EMMEANS command (at least I did not find an easy way).

Now, in models with many factors it is a bit tricky to write down the /TEST line, the sequence of 1/2, 1/4 etc, called L matrix. Typically if you get the error message: "the L matrix is not estimable", you are forgetting some elements. One link that explains the receipt is this one: https://stats.idre.ucla.edu/spss/faq/how-can-i-test-contrasts-and-interaction-contrasts-in-a-mixed-model/

Answer 2

I don't know SPSS syntax particularly well, but, if I understand your situation correctly, the significant interaction means that, in order to adequately assess the significance of your main effects, you'll need to do separate analyses. I think the best way to proceed is to do separate repeated measure analyses for each level in your grouping factor. Perhaps someone else can speak better to the question of how to handle correcting for multiple comparisons during post-hoc analysis, but I'm pretty sure you still need to use a correction. You might try Tukey's, as a multiple comparison correction!

Answer 3

In short. There is no globally accepted convention for these situations. Some will use Bonferroni corrections. Some will force the Tukey HSD framework to dance for them (e.g. Maxwell & Delaney). In contrast...

COMPARE(time) ADJ(BONFERRONI)", just after "/EMMEANS=TABLES(newgroup*time)

... does seem to use the Bonferroni correction. However, this approach is likely conservative, especially in the face of Holm-Sidak style corrections. (ESPECIALLY if you don't use the MSW as the error term for your post-hoc comparisons).