2

I am not sure if I've made a computational error, am failing to think something through (too late in the day), or just really don't understand confidence intervals as much as I was hoping I did, but I ran a linear model with Condition as a predictor (2 groups) and the slope representing the group difference was significantly different from zero:

b = 0.10599, t = 2.043, p = 0.045, 95% CIs [0.0023, 0.21]

However, when I compute the CIs for the two group means they overlap: 

Experimental: M = .32, 95% CIs [.24, .39]

Control: M = .21, 95% CIs [.14, .28].

What's going on? The overlapping CIs suggest to me that the means aren't significantly different, but I thought testing whether the mean difference differs significantly from zero is the same as testing whether the means differ significantly from each other?

PanPsych
  • 440
  • 4
  • 13

1 Answers1

6

See this page for an explanation of the relation between overlapping confidence intervals and significance tests for the difference between 2 groups. No overlap of 95% confidence intervals is much too stringent a test for the difference between the two means at p = 0.05. Each mean value would be in the space outside its own 95% CI only a fraction of 0.05 of the time in repeated samplings. Crudely, requiring lack of overlap of the two 95% might be thought of as requiring this restriction to hold simultaneously for both mean values, which would occur only (0.05)^2 = 0.0025 fraction of the time. More detailed calculations on the linked page show that non-overlap of 95% CIs is about equivalent to a p = 0.005 test for the difference between the 2 means, under reasonable assumptions.

Community
  • 1
EdM
  • 57,766
  • 7
  • 66
  • 187
  • Thanks very much! I'll bet that this is one reason why people tend to use SE bars instead of CIs for bar charts! – PanPsych Mar 26 '16 at 13:26
  • 1
    Those who tend to use bar charts a lot probably haven't thought much about what error bars to display, other than that SEs are smaller. Showing non-overlap of 83% CI would approximately be equivalent to p = 0.05 for a difference in means according to the page I linked in my answer, but I'm unaware of any groundswell of support for such displays. – EdM Mar 26 '16 at 13:49
  • I've heard Geoff Cummings talk about shifting towards using CIs instead of null hypothesis testing, and I believe he and others have argued that CIs should be displayed in bar charts instead of SEs. I would do this if there was a good reason to, but I worry that others may get confused as I did. (I could avoid the bar chart altogether but for some reason people like to see them.) – PanPsych Mar 26 '16 at 18:01