2

I'm a statistics noob, and I understand this is a very simple question, but I didn't know what to search for to find an answer.

I have paired values for a variable (pre and post treatment) for two groups (treatment and control). Using a paired samples t-test for each group, I have found a statistically significant change in the treatment group but not in the control group. I have also used a t-test on the difference between the change in the two groups, but that is not significant.

How can that be? How can one group be significant and one not, but still the difference between the change in the groups is not significant?

whuber
  • 281,159
  • 54
  • 637
  • 1,101
HaBo
  • 23
  • 4

2 Answers2

2

This happens all the time in statistics. Just because estimate A is significantly different from zero, and estimate B is NOT significantly different from zero, doesn’t mean A and B are significantly different from each other.

This is a bit of an over-simplification (which I’ll explain in a second) but it might help to think about this in terms of confidence intervals.

Let’s say you estimate that, between time 1 and time 2, the treatment group improved by 4 points with 95% confidence intervals ranging from 2.5 to 5.5. The confidence intervals don’t overlap with zero so this increase is statistically significant at the 95% level.

Then you estimate that the control group improved by 2 points with 95% CIs ranging from -1 to 4. This confidence interval overlaps with zero so this change is not significant at the 95% level.

So you have one change that is significantly different from zero, and another that is not. But notice that the confidence intervals around these two estimates overlap with each other: the treatment group might have only increased by 2.5, which is right in the middle of the CIs for the treatment group. So the difference between the differences (the “difference-in-differences” which is the kind of analysis you are running) is not itself statistically significant.

The oversimplification here is that you can’t actually tell whether the difference between two estimates is statistically significant at the 95% level just by checking whether the 95% confidence intervals overlap (if two CIs don't overlap you can be sure that the difference is significant at that level of confidence but if they do overlap slightly the difference might still be significant because of stuff related to how you calculate standard errors) but this is one of the reasons why t tests (which account for this stuff) exist in the first place!

Graham Wright
  • 1,559
  • 1
  • 11
  • See also https://stats.stackexchange.com/questions/21722/confidence-interval-overlap-judging-the-similarity-of-estimates, https://stats.stackexchange.com/questions/48060/how-to-interpret-the-overlap-or-non-overlap-of-two-related-confidence-intervals – kjetil b halvorsen Dec 16 '20 at 16:32
1

Nothing too strange happening here. Consider the following normal distributions

$$TreatmentPre \sim \mathcal{N}(\mu=4.5, \sigma^2=1)$$ $$TreatmentPost \sim \mathcal{N}(\mu=5.5, \sigma^2=1)$$ $$ControlPre \sim \mathcal{N}(\mu=4.9, \sigma^2=1)$$ $$ControlPost \sim \mathcal{N}(\mu=5.1, \sigma^2=1)$$

The difference between means of treatment pre-post is much larger than that for control, but the total means for treatment and control are the same. I have tried this for 100 sample points for each of the 4 categories and only treatment pre vs post came out significant, the difference across groups was not.

The simplest explanation for the phenomenon that you observe is that there is indeed also a difference between treatment and control groups (when considering only pre or only post treatment), but you just barely don't have enough datapoints to demonstrate that the difference is significant, whereas you have enough datapoints to demonstrate the effect of the treatment, which happens to be stronger

Aleksejs Fomins
  • 1,499
  • 3
  • 18