2

I'm trying to perform a meta-analysis where my effects are mainly one-sample t-test against chance level. I've easily found the one-sample cohen's d formula but I'm not able to find the variance formula to weight the estimated effect size. I've found this post however I would like to have a clear reference or maybe other suggestions. Thanks!

Filippo Gambarota
  • 181
  • 4
  • Could you elaborate on what this variance would be of? If it's the estimated effect size in a given experiment, then that's precisely what appears in the numerator of the Student $t$ statistic and an estimate of its variance (adjusted by a factor of $1/\sqrt{n}$) appears in the denominator. – whuber Jun 23 '20 at 12:20
  • I would like to compute the variance of the effect-size to perform the inverse-variance weighting for the meta-analysis. For example from the book of Borenstein (2009) for each effect size measure (not the one-sample cohen's d) there is the formula for the variance also. – Filippo Gambarota Jun 23 '20 at 12:53

1 Answers1

1

First off, it is important to clarify the nature of your effect size. There are two ways to standardized the mean difference. The first is with either the time 1 standard deviation or the pooled time 1 and time 1standard deviations. The second is with the standard deviation of the gain scores. The relationship between these two denominators is $$ s = \frac{s_{gain}}{\sqrt{2 \left( 1 - r \right)}} $$ where $s$ is the pooled or raw standard deviation and $r$ is the correlation between the time 1 and time 2 scores. I strongly recommend standardizing on $s$ and not $s_{gain}$. It is not clear how to interpret the latter as you are standardizing the mean change on the variability in change.

Assuming you are standardizing the mean difference using $s$, then the variance for the effect size is $$ v = \frac{2*(1-r)}{n} + \frac{d^2}{2*n} ~.$$ The details for this are worked out in:

Becker, B. J. (1988). Synthesizing standardized mean‐change measures. British Journal of Mathematical and Statistical Psychology, 41(2), 257-278.

The challenge, of course, is that in the context of meta-analysis, the studies often do not report $r$. Some meta-analysts use a guesstimate of $r = .5$ and you can do sensitivity analysis with different values. If the study reports the time 1 and time 2 standard deviations and means and one-sample t-test, then you can recover $r$ using the following: $$ r = \frac {\left(s_1^2 t^2 + s_2^2 t^2 \right) - \left(\overline{x}_2 - \overline{x}_1\right) n} {2 s_1 s_2 t^2} ~.$$ This assumes that the sign of $t$ matches the sign of the mean difference. Flip one or both as needed.

dbwilson
  • 1,543
  • 7
  • 10
  • Thanks for your detailed explanation and for the reference. However, I think that I'm missing something. What are Time 1 and Time 2? In my experiments the performance, let's say 58% (SD=0.03) accuracy is compared to the chance level. So I can compute the cohen's d $d=\frac{\bar x - chance\ level}{s}$. I don't have two measures for the subject. – Filippo Gambarota Jun 25 '20 at 16:54
  • I had assumes two repeated measures or paired data, as is typical of a dependent t-test. I misread your post. Sorry. You have a one-sample t-test with an non-zero null. The inverse variance is simply 1/n (the standard error of your d is simply 1/sqrt(n). – dbwilson Jun 29 '20 at 12:32
  • thanks for your time! so the formula inside this [post](https://stats.stackexchange.com/questions/226836/sampling-variance-for-meta-analysis-one-sample-data) is not correct right? – Filippo Gambarota Jun 29 '20 at 14:54