3

I am trying to calculate the effect size of exercise on depression. So I have 2 groups (control and intervention) in an RCT setting. I have both mean values and standard deviation of groups at T0 (baseline) and Tf (final). The number of participants in each group is different due to dropouts etc., hence the variations.

My question is how should I calculate an effect size out of this setting? Intuitively, I feel like I need to standardize every mean value first, calculate SMD (standardized mean difference) between Tf and T0 for each group, and compare (take SMD again), this time for the difference between groups. However, I am not sure how should I take the variance into account. Should I just use the variance to standardized each value of the group? But, how I am going to estimate the variance of the effect size?

Kuku
  • 1,013
  • 5
  • 18
yer
  • 115
  • 5
  • Is depression measured in the same way for all studies? You would not use SMD for a prepost control design, but a (standardized) mean change effect size (or, optimally, an ANCOVA effect estimate). – Kuku May 19 '21 at 15:57
  • They are not that's actually why I feel like need to use SMD – yer May 19 '21 at 17:34

1 Answers1

4

Your setting describes a pretest-posttest-control group design, hence you should not use a (standardized) mean difference measure but a (standardized) measure of mean change. You can read a broad overview in Chapter 6 of the Cochrane Handbook and in Viechtbauer's documentation on conducting meta-analysis in R with the metafor package.

What is mean change? This is a non-trivial problem, in close connection with Lord's paradox. You have roughly three options (following Senn's and McKenzie et al.'s, 2015 taxonomy):

  • Simple Analysis of Final Values (SAFV): $\bar{Y}^{Int}_{Tf} - \bar{Y}^{Con}_{Tf} $
  • Simple Analysis of Change Score (SACS): $(\bar{Y}^{Int}_{Tf} - \bar{Y}^{Int}_{T0}) - (\bar{Y}^{Con}_{Tf} - \bar{Y}^{Con}_{T0}) $
  • ANCOVA Effect Estimate: $(\bar{Y}^{Int}_{Tf} - \bar{Y}^{Con}_{Tf}) - \hat{\beta}(\bar{Y}^{Int}_{T0} - \bar{Y}^{Con}_{T0}) $

You can see McKenzie's et al. paper for a thorough comparison and formulas for the computation of the variances of measures. The general advice is to prefer the ANCOVA effect estimate, as both SAFV and SACS tend to be biased.

Regarding the variance, for both SACS and ANCOVA you need to have an estimate of the correlation between the pre and post measures (or, equivalently, the standard deviation of the change score), which is usually not reported. Once you have one of these values, you are ready to estimate the variance of the effect size, since $r = \frac{SD^2_{T0} + SD^2_{Tf} - SD^2_{Change}}{2*SD_{T0}*SD_{Tf}}$ (Fu et al., 2008, p.15). In some cases, you may have the confidence interval of the change score, which is also sufficient following the indications in the Cochrane Handbook (7.7.3.2.).

Note

  1. The options are summarized as three here, but you can read more on different ways to set up the ANCOVA effect estimate and performance comparisons in O'Connell et al., 2017.
  2. If the pre-post correlation is larger than 0.5, SACS is prefered over SAFV (and vice-versa, Fu et al., 2008, p.9).
Kuku
  • 1,013
  • 5
  • 18