0

I have a maybe really silly question about t-test. I know that t-test is a test for comparing means between two sets of data. But I also was taught that when 2 SD cross over with each between the data (ie 2 (mean +/- 1 SD) ranges overlap), it means there is no statistical difference in between two groups. But my result was different than what I was expected. Two mean +/- 1 SD overlap with each other (ex mean1= 28.01, mean2=28.96 and the SDs are about 10 for each mean) but has p-value at 0.003. Can someone tell me how did this happen and what should I pay attention to?

ps. N>30K and the data is heteroscedastic.

Willie
  • 49
  • 4
  • It sounds like the t-test is comparing each mean with zero rather than to each other. – HStamper Apr 21 '17 at 21:51
  • (1) You appear to confuse the SD with the SE. (2) Even when the SE ranges overlap, the difference can be significant: see https://stats.stackexchange.com/a/31660/919 for some remarks and https://stats.stackexchange.com/a/18259/919 for a quantitative analysis of how much overlap can be considered significant. – whuber Apr 21 '17 at 22:42

1 Answers1

0

Hm... I'm not an expert, but your t-test result doesn't seem all that surprising to me. If we assume that your samples are in fact drawn from different populations (let's say with $\mu_1 = 28.01, \sigma_1 = 10$ and $\mu_2 = 28.96, \sigma_2 = 10$) then it's entirely possible that both

  1. their SDs could overlap, and
  2. a t-test could correctly detect the difference between the two group means.

Yes, the difference is small, but 30K points is an awful lot of data, and more data makes the t-test more sensitive.

But I also was taught that when 2 SD cross over with each between the data (ie 2 (mean +/- 1 SD) ranges overlap), it means there is no statistical difference in between two groups.

^ Are you sure this is correct?

If someone else can give a more rigorous answer, please do!

dB'
  • 225
  • 3
  • 15
  • 1
    dB has likely diagnosed the core issue here: even though the difference in means is small compared with the SD, you have enough data that you can be quite sure that the difference is real. The relevant comparison is not the number of SDs that separate the means, but the number of SEMs that separate the means. SEM = SD / sqrt(N), so you know the means very well, within about .08, and .08 >> (28.96 - 28.01). – David Wright Apr 21 '17 at 22:08