3

If you have five groups and you wish to know if there is a significant difference between any of the group means, you would have to do 10 pairwise comparisons to test all possible pairs of means. You would have to test

  Mean for group 1 vs. mean for group 2 
  Mean for group 1 vs. mean for group 3 
  Mean for group 1 vs. mean for group 4 
  Mean for group 1 vs. mean for group 5 
  Mean for group 2 vs. mean for group 3 
  Mean for group 2 vs. mean for group 4 
  Mean for group 2 vs. mean for group 5 
  Mean for group 3 vs. mean for group 4 
  Mean for group 3 vs. mean for group 5 
  Mean for group 4 vs. mean for group 5

If you do each of these tests at the alpha = .05 level, you have 5 chances in a hundred of concluding the difference is significant when it is really just due to chance. If you 10 such tests you have 10 times .05 = .50 or a 50/50 chance that one will be significant just by chance. This is an unacceptable error rate. We could use a Bonferroni correction to adjust the significance levels we're hoping to detect such that we won't have a false Type I error being alerted.

Alternatively, we could also use ANOVA to compare all of these groups.

Aside from computational limitations, what advantage does ANOVA have over pairwise Bonferroni corrected t-tests?

Nick Cox
  • 48,377
  • 8
  • 110
  • 156
  • I guess, this answers your question http://stats.stackexchange.com/questions/9751/do-we-need-a-global-test-before-post-hoc-tests/9753#9753 – peuhp Mar 21 '16 at 19:59
  • 1
    There is actually $1-(1-0.05)^{10}\approx 0.40$ chance that at least one will be significant under the null, not $0.5$. But it does not change the rest of your question, +1. – amoeba Mar 21 '16 at 20:14
  • 1
    One hint is that your 10 comparisons are not independent. – amoeba Mar 21 '16 at 21:31
  • One guess that I can make with that hint is that ANOVA bases it's analysis on a "grand mean" - that may preserve the independence between comparisons. But I'm honestly not quite certain. –  Mar 22 '16 at 16:38
  • 1
    See [Relation between omnibus test and multiple comparison?](http://stats.stackexchange.com/q/59910/17230). Part of the answer is that omnibus tests like ANOVA are more sensitive to many medium-sized deviations from the null than a collection of multiplicity-corrected pairwise tests; conversely ANOVA's less sensitive to a single large effects, or a few quite large effects, when the others are small. – Scortchi - Reinstate Monica Mar 22 '16 at 17:22

0 Answers0