2

I'm reviewing some meta-analyses on psychotherapies and trying to make sense of the statistics related to heterogeneity, specifically Cochran's Q and the I-squared value.

I'm confused about how to interpret a non-significant Q statistics while the I-squared value is >40%. Reading Cochran advice, anything I-squared value above 40% is potentially problematic but if the Q statistic is not significant then what does that mean in terms of describing whether heterogeneity (a) exists and (b) warranted, for example, carrying out a moderator analysis.

For example, the stats from one meta-analysis are below:

Q(2) = 4.05, p > 0.05, I2 = 50.66

The Q is non-significant but the I-squared is >40%. The authors state that the heterogeneity was "low". While the number of studies is extremely small (n=3) and a moderator analysis not really possible from my reading, how would the heterogeneity in this meta-analysis be best described?

Strooby
  • 83
  • 6

2 Answers2

2

Cochran's Q-test is known to have low statistical power if only a small number of effect sizes are included in a meta-analysis (see the refs below for literature on this). Hence, it is perfectly possible that the Q-test is nonsignificant whereas the $I^2$-statistic is quite large. It is important to keep in mind that the $I^2$-statistic itself is imprecise as well, so it is generally advised to construct a confidence interval around the $I^2$-statistic to illustrate its imprecision. Multiple methods to compute these confidence intervals are, for instance, implemented in the R package metafor.

References

Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. British Medical Journal. 2003;327(7414):557-560.

Hardy RJ, Thompson SG. Detecting and describing heterogeneity in meta-analysis. Statistics in Medicine. 1998;17(8):841-856.

Viechtbauer W. Hypothesis tests for population heterogeneity in meta-analysis. British Journal of Mathematical and Statistical Psychology. 2007;60:29-60.

User33
  • 307
  • 1
  • 7
2

It is easiest to see what is going on here if you compute a confidence interval for $I^2$. By my calculations with $Q=4.05$ and 3 studies the 95% interval for $I^2$ is from 0% to 86% implying that we really have next to no information about $I^2$ with such a small sample of primary studies.

It is also worth noting that relying on arbitrary cut-offs for $I^2$ or indeed any statistic is fraught with problems. There are areas of science where I would be very surprised if there was not substantial heterogeneity. It is also the case that if the primary studies are very large and give rise to precise estimates then since $I^2$ compares within study to between study variability the value of $I^2$ is almost bound to be large. See the paper by Rücker and colleagues entitled "Undue reliance on $I^2$ in assessing heterogeneity may mislead"

mdewey
  • 16,541
  • 22
  • 30
  • 57