2

I have a paired data set which I have placed into $x$ and $y$ columns where $x$ are the control values and $y$ are the values following drug treatment. $N=10$ for both $x$ and $y$ columns as they are paired data. Each $x$ is the control for the corresponding $y$.

I have seen in various texts stating that when standard error margins overlap, the data cannot significant. By standard error margin, I am referring to ($SE_\bar x = SD/\sqrt N$).

However, I have conducted two-tailed paired $t$-tests on my data set (comparing the means of all values in $x$ versus the means of all values in $y$) and my results yield statistical significance with a $p$-value $< 0.05$ (despite there being overlapping standard error margins with data in $x$ and $y$).

My question is: in a paired data set, is it possible for there to be statistical significance between the control ($x$) and drug treatment ($y$) despite having overlapping standard errors? My $t$-test was done using GraphPad prism so I'm confident there are no errors in the $t$-test.

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
Provo
  • 23
  • 1
  • 4
  • Yes, there is. The standard error of the difference (which is what you care about) is dependent on the correlation between the measures. – Jeremy Miles Aug 05 '15 at 00:10
  • I think you mean overlapping *confidence intervals* and not overlapping standard errors, as the latter are just numbers and can't overlap. – dsaxton Aug 05 '15 at 00:19
  • @dsaxton, he's referring to a plot of the condition means w/ $\pm 1$ SE error bars. The ends of those error bars are overlapping. – gung - Reinstate Monica Aug 05 '15 at 00:50
  • 1
    I wouldn't be so dismissive of the content of your data / your concrete situation. You would be surprised to know how relevant that can be to simple "statistical questions". – gung - Reinstate Monica Aug 05 '15 at 00:56
  • I had a friend in my dept look at my data. He commented that "the large overlapping error bars of the control and drug groups makes me unconvinced that the data is significant". Not sure if he is making a valid point or not. After all, the t-test confirmed there was a significant difference despite overlapping SE bars. – Provo Aug 05 '15 at 01:07

1 Answers1

5

Yes, it's quite possible for the $\pm 1$ SE error bars to overlap, but still have a significant pairwise $t$-test. The reason is that your error bars are calculated on the between subjects data, but the test is of the within subjects data. They are not the same thing, so they don't have to be consistent with each other.

It is fine to calculate and present pre- and post-treatment means and error bars, but it can lead to just this confusion. To understand what a paired $t$-test is doing, you need to think of it as a one-sample $t$-test on the differences. That is, first subtract the pre-treatment value from the post-treatment value for each patient. Then you have only one variable: difference scores. Then you calculate the mean and standard error of those. The pairwise (one-sample) $t$-test is the mean divided by the SE. If you like, you can even make a bar plot with a single forlorn-looking bar representing the mean difference with it's corresponding error bar. That is the actually the relevant plot for a pairwise $t$-test. If the test was significant, the error bar will not overlap $0$. To see this concretely, see my answer here: Is using error bars for means in a within-subjects study wrong?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
  • This is great answer, but I am a little confused, Is there possible that $n$ is different when they calculate the SE? Also if there are different groups for a repeated measurements the within data and between data may change. In fact, I also encounter a same situation, the error bars are overlapped but the difference is significant for a repeated measurement. I may post a question later. – Deep North Aug 05 '15 at 01:44
  • To give an example : suppose I have 10 adults and I want to measure their blood sugar before/after treatment with Drug A. From the pre-drug blood sugar data, we see a wide variation in the scores from the 10 ppl and thus the SE of the control data is large. After treatment, we take their blood sugar again. After drug treatment, all of the patients showed a decrease in blood sugar compared to their own pre-drug levels. However, there was still a large variation in the values of the 10 people after drug treatment. If +/- SE bars overlap in this scenario does this mean my data is not significant? – Provo Aug 05 '15 at 02:24
  • 1
    @Provo, what a *paired* t-test assesses are the differences. So if the pre & post variability are sufficiently large for their (ie the between subjects data) error bars to overlap is irrelevant to the significance of the t-test. – gung - Reinstate Monica Aug 05 '15 at 02:31
  • Do you know of a referece (peer-reviewed publication) that would support the statment that error bars of within-subjects data has nothing to do with the P-value of paired data? – Provo Aug 05 '15 at 22:30
  • @Provo the *within-subjects* error bars certainly have a lot to do with the p-value from a paired t-test. It is the *between-subjects* error bars that don't. At any rate, I don't know of a reference that says that & I doubt there is one--it's really obvious (to statisticians). I guess you could cite a stats 101 textbook. – gung - Reinstate Monica Aug 05 '15 at 22:56