2

Am I right to say that the results from the a repeated measures ANOVA does not tell you where a difference is in the group (if any)? So a post hoc test needs to be carried out? I'm using Stata to analyze my data.

Below is the typical result I got, but results are significant. How do I perform a further analysis to find out which group had a significant difference?

enter image description here

Alexis
  • 26,219
  • 5
  • 78
  • 131
Emma Dixon
  • 87
  • 1
  • 8

2 Answers2

2

That is correct. The repeated measures ANOVA is an omnibus test, so if you reject its null hypothesis, then you need to proceed to pairwise comparisons (you do not need to proceed to pairwise tests otherwise). In Stata following a repeated measures ANOVA you would use the test post-estimation command. To see Stata's help on this topic within Stata type: help anova postestimation##test.

This is also where this issue of multiple comparisons arises and about which you can read more in the referenced Stata documentation. Note that if you wish to perform multiple comparisons adjustments based on the false discovery rate, as opposed to the family-wise error rate, you will need to use the no adjust option for test along with the qqvalue package (within Stata type net sj 10-4 st0209).

Community
  • 1
Alexis
  • 26,219
  • 5
  • 78
  • 131
  • A good answer. I do think it bears mentioning that you don't traditionally follow up a null result with post-hoc comparisons however. – Marcus Morrisey Jun 16 '14 at 17:40
  • @MarcusMorrisey Already implied in the second clause of the second sentence. – Alexis Jun 16 '14 at 17:51
  • 1
    That's true. I simply thought it might be helpful for Emma to have it laid out a little more explicitly since her question indicates that she is likely a beginner with statistical analysis. In any case, I think our comments should make that sufficiently clear. Cheers. – Marcus Morrisey Jun 16 '14 at 17:58
0

Within subjects anova tests an omnibus null hypothesis of the form:

$ \mu_1 = \mu_2 = ...= \mu_k $

(where k is the number of levels for your factor) against the alternative that at least one mean is different. So, you are correct in saying that you cannot identify which mean or means are different from each other based only on the anova results.

You will need to follow up with post-hoc comparison procedures, very often pairwise t-tests (although there are other options). Exactly what tests are interesting or required is going to depend on your specific research question. You will have to keep in mind the risk of alpha inflation when performing multiple tests. There are a number of procedures to adjust for this risk. Wikipedia has a reasonable summary of some the more common procedures here. I am not familiar with stata so I cannot offer any advice on how these may be implemented in that package.

In reference to the results you are displaying, however, I note that your anova indicates that you should not reject the null hypothesis. In most disciplines an $ \alpha$ of .05 or lower is considered the minimum criterion for statistical significance (although this approach is problematic, it is highly conventional. For additional information this is a decent place to start). Your p-value is around .15. By convention, one does not usually follow up a null result with any additional tests (although it is possible that more focused tests could reveal significant differences).

Marcus Morrisey
  • 1,056
  • 8
  • 20
  • 1
    This "Your test is indicating no effect." is not quite accurate, and veers towards the fallacy of accepting the null hypothesis. Evidence for no effects would require equivalence tests. – Alexis Jun 16 '14 at 17:53
  • A fair point. I was attempting to keep complexity at a reasonable level for someone who likely has limited experience with statistics. I do think that my caveat afterwords implies that this was not a case of accepting the null. Nonetheless, in the interest of clarity, I'll edit that out. – Marcus Morrisey Jun 16 '14 at 18:03