2

I'm expecting Groups 1 & 2 to be very similar and Group 3 to be different.

I'm using a very large data set and feel that power won't be an issue. Initially I felt that I'd then be expecting a significant ANOVA, and then in planned comparisons I'd expect 1 vs 2 to be non-significant while 1 vs 3 and 2 vs 3 should be significant. While that is actually how my data turned out I'm now thinking this was the wrong way to go about things.

When comparing 1 vs 2 should I have used equivalence testing (e.g. http://www.cscu.cornell.edu/news/statnews/stnews85.pdf) instead? If so, how could I have gone about that using SPSS or some other commonly available software?

user1205901 - Reinstate Monica
  • 11,303
  • 26
  • 77
  • 152
  • 1
    Since you haven't pre-specified equivalence regions, I'm not so sure about standard (Confidence Interval- or Two One-Sided T-Test based) equivalence testing. You could possibly use a Bayesian hypothesis test that can "accept the null", such as this thing which allows you to just enter the t value, n, and press the button: http://pcl.missouri.edu/bayesfactor Alternatively, you could compute the 95% CI of the difference between 1 and 2 in standardised effect sizes (Cohen's d). If it only contains "small" values, you may say with some confidence that 1 and 2 are unlikely to differ "much". – jona Sep 18 '13 at 12:40
  • 1
    You may want to consider a pair of contrasts: $\mu_3 - (\mu_1+\mu_2)/2$ and then $\mu_1 - \mu_2$. If those are what you're interested in, you don't really need the ANOVA itself; they'll partition the SS for the ANOVA anyway, so you can just test those things directly. – Glen_b Sep 19 '13 at 05:13
  • @jona Thanks for the advice. In my data the effect sizes were not expected to be especially large even in the comparisons in which differences were expected. Should I therefore 'massage down' the Cohen's guideline for what counts as "small"? – user1205901 - Reinstate Monica Sep 19 '13 at 05:37
  • @Glen_b Thanks I can see the sense in that. Do I understand correctly that I'd be predicting no significant difference in the $\mu_1 - \mu_2$ contrast, but other than that would have no specific prediction about what would happen? – user1205901 - Reinstate Monica Sep 19 '13 at 05:44
  • Well, perhaps, but it looks to me like you made a specific prediction for both. In any case your predictions/research-hypotheses don't change the null you're testing. However, if your predictions make sense you'd tend to see small effect size in the $\mu_1-\mu_2$ contrast and a (relatively) larger effect size in the other contrast. That may or may not translate into significance, which rather depends on your sample size as well. With small samples, it may be that there are no differences while at larger sample sizes even trivial effects may be statistically significant. – Glen_b Sep 19 '13 at 07:18

0 Answers0