How to best handle subscores in a meta-analysis?

Question

I am conducting a meta-analysis of effect sizes d in R using the metafor package. d represents differences in memory scores between patients and healthy. However some studies report only subscores of the measure of interest d (e.g. several different memory scores or scores from three separate blocks of memory testing). Please see the following dummy data set with d representing the studies' effect sizes as well as their standard deviations sd.:

d <- round(rnorm(5,5,1),2)
sd <- round(rnorm(5,1,0.1),2)
study <- c(1,2,3,3,3)
subscore <- c(1,1,1,2,3)
my_data <- as.data.frame(cbind(study, subscore, d, sd))

library(metafor)
m1 <- rma(d,sd, data=my_data)
summary(m1)

I would like to ask for your opinion for the best way how to handle these subscores - e.g.:

1. Select one subscore from each study that reports more than one score.
2. Include all subscores (this would violate the assumption of independency of the rfx model as subscores of one study come from the same sample)
3. For each study that reports subscores: calculate an average score & average standard deviation and include this "merged effect size" in the rfx meta-analysis.
4. Include all subscores and add a dummy variable indicating from which study a certain score is derived.

Answer 1

This type of data is known as the dependent effect sizes. Several approaches can be used to handle the dependence. I would recommend the use of three-level meta-analysis (Cheung, 2014; Konstantopoulos, 2011; Van den Noortgate et al. 2013). It decomposes the variation to level 2 and level 3 heterogeneity. In your example, the level 2 and level 3 heterogeneity refer to the heterogeneity due to subscales and studies. The metaSEM package (http://courses.nus.edu.sg/course/psycwlm/Internet/metaSEM/) implemented in R provides functions to conduct three-level meta-analysis. For example,

## Your data
d <- round(rnorm(5,5,1),2)
sd <- round(rnorm(5,1,0.1),2)
study <- c(1,2,3,3,3)
subscore <- c(1,1,1,2,3)
my_data <- as.data.frame(cbind(study, subscore, d, sd))

## Load the library with the data set  
library(metaSEM)
summary( meta3(y=d, v=sd^2, cluster=study, data=my_data) )

The output is:

Running Meta analysis with ML 

Call:
meta3(y = d, v = sd^2, cluster = study, data = my_data)

95% confidence intervals: z statistic approximation
Coefficients:
            Estimate  Std.Error     lbound     ubound z value  Pr(>|z|)    
Intercept 4.9878e+00 4.2839e-01 4.1482e+00 5.8275e+00  11.643 < 2.2e-16 ***
Tau2_2    1.0000e-10         NA         NA         NA      NA        NA    
Tau2_3    1.0000e-10         NA         NA         NA      NA        NA    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Q statistic on homogeneity of effect sizes: 0.1856967
Degrees of freedom of the Q statistic: 4
P value of the Q statistic: 0.9959473
Heterogeneity indices (based on the estimated Tau2):
                              Estimate
I2_2 (Typical v: Q statistic)        0
I2_3 (Typical v: Q statistic)        0

Number of studies (or clusters): 3
Number of observed statistics: 5
Number of estimated parameters: 3
Degrees of freedom: 2
-2 log likelihood: 8.989807 
OpenMx status1: 1 ("0" and "1": considered fine; other values indicate problems)

In this example, the estimates of the level 2 and level 3 heterogeneity are close to 0. Level 2 and level 3 covariates may also be included to model the heterogeneity. More examples on the three-level meta-analysis are available at http://courses.nus.edu.sg/course/psycwlm/Internet/metaSEM/3level.html

References

Cheung, M. W.-L. (2014). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods, 19(2), 211-29. doi: 10.1037/a0032968.

Konstantopoulos, S. (2011). Fixed effects and variance components estimation in three-level meta-analysis. Research Synthesis Methods, 2(1), 61–76. doi:10.1002/jrsm.35

Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods, 45(2), 576–594. doi:10.3758/s13428-012-0261-6

Answer 2

I agree it's a tricky situation. These are just a few thoughts.

Whether to average d effect sizes: If you're not interested in subscales, then my first choice would be to take the average effect size for the subscales in a given study.

That assumes that all subscales are equally relevant to your research question. If some scales are more relevant, then I might just use those subscales.

If you are interested in differences between subscales, then it makes sense to include the effect size for each subscale coded for type.

Standard error of d effect sizes: Presumably you are using a formula to calculate the standard error of d based on the value of d and the group sample sizes. Adapting this formula, we get

$$se(d) = \sqrt{\left( \frac{n_1 + n_2}{n_1 n_2} + \frac{d^2}{2(n_1+n_2-2)}\right) \left(\frac{n_1 + n_2}{n_1+n_2-2} \right)}, $$

where $n_1$ and $n_2$ are the sample sizes of the two groups being compared and $d$ is Cohen's $d$.

I imagine you could apply such a formula to calculate the standard error to the average d value for the subscales.