6

Is it possible to statistically compare the Cohen's $d$ of two studies in order to determine of one of the two Cohen's $d$s is statistically significantly larger (or smaller) than the other one?

I have calculated Cohen's $d$ for two virtually identical studies. In one study the cohen's $d$ for control group vs. treatment group is $1.05$; in the other study it is $1.31$. In the second study Cohen's $d$ is larger than in the first study. But can I perform a test to determine if the difference is statistically significant? If a statistical test is not possible is there a method / guidelines that would allow me to say if a difference of $0.26$ between the two Cohen's $d$s is small, medium or large (in other words relevant)?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
user35129
  • 61
  • 1
  • 3
  • 1
    This is a standard topic in meta-analysis. Note that Cohen's d is biased & needs to be adjusted. You need to know the sample sizes (n's) for this. If you have that, the bias-adjusted standardized mean difference has a variance that is a function of the statistic, so you only need the statistic to form (eg) confidence intervals. In addition, there are chi-squared tests in MA to determine if BASMD's vary more than they 'ought to'. Do you know the constituent n's? – gung - Reinstate Monica Nov 21 '13 at 18:13
  • Thank you for your answer! Yes, I know the sample sizes of both studies (I ran one study and the other one is a published study) – user35129 Nov 22 '13 at 10:57
  • I have calculated the d unbiased according to Hedges and Olkin (1985). The chi-square test you are referring to would be the statistic Q? – user35129 Nov 22 '13 at 13:32
  • Yes, I believe that's what they call it. I think it should be possible from the CIs alone, but the chi-squared test is the standard way. – gung - Reinstate Monica Nov 23 '13 at 00:28

2 Answers2

7

If $d$ is the observed Cohen's d value, then the sampling variance of $d$ is approximately equal to:

$$v = \frac{1}{n_1} + \frac{1}{n_2} + \frac{d^2}{2(n_1+n_2)}.$$

So, to test $H_0: \delta_1 = \delta_2$ (where $\delta_1$ and $\delta_2$ denote the true d values of the two studies), compute:

$$z = \frac{d_1 - d_2}{\sqrt{v_1 + v_2}},$$

which follows approximately a standard normal distribution under $H_0$. So, if $|z| \ge 1.96$, you can reject $H_0$ at $\alpha = .05$ (two-sided).

As mentioned by gung, you could consider applying the bias-correction first, but unless sample sizes are small, the impact on $z$ will be negligible.

Wolfgang
  • 15,542
  • 1
  • 47
  • 74
  • 1
    Thanks for this. I thought there was a way to do it directly. In retrospect, this is pretty obvious, but you aren't usually working w/ just 2 $d$s & wonder if they're heterogeneous. – gung - Reinstate Monica Nov 25 '13 at 15:55
  • 1
    If you do a regular meta-analysis on those 2 $d$ values, then the $Q$ test for heterogeneity will just be the square of the $z$ statistic above. So, those two approaches are exactly equivalent. – Wolfgang Nov 25 '13 at 16:46
  • 2
    Excuse me for the ignorant question, but where do these formulas come from? As in, how did you derive them? Second question: if you're comparing effect sizes from two separate samples, why not just do an unpaired t-test, using the d's as the means, and the standard error of the d's (square root of the variances) as the SEMs? Or would you use them as the SD's? Basically, is there any way to get a p value in addition to confidence intervals? – Jake Moskowitz Feb 17 '15 at 02:28
1

I think I have found the source of the formula in the book of Borenstein et al (2009) Introduction to meta-analysis. John, Wiley & Sons Ltd. It is on p.156 and is used to compare two subgroups. Another option could be to take a look at the other hetereogeneity metrics described in the book. Good luck.

RosannaResearch
  • 21
  • 2