When/why does the central tendency of a resampling simulation markedly differ from the observed value?

Question

Should one always expect the central tendency (i.e., mean and/or median) of a bootstrapped sample to be similar to the observed value?

In this particular case I have responses that are distributed exponentially for subjects across two conditions (I didn't run the experiment, I only have the data). I have been tasked with boot strapping the effect size (in terms of Cohen's d, the one-sample formula, i.e. $\bar{M_D}\over{s_D}$ where is the sample estimate of the population standard deviation. The forumla for this is provided in Rosenthal & Rosnow (2008) on pg 398, equation 13.27. They use $\sigma$ in the denominator because it is historically correct, however standard practice has misdefined d as using $s$, and so I follow through with that error in the above calculation.

I have randomized both within participants (i.e. a participants RT may be sampled more than once) and across subjects (participants may be sampled more than once) such that even if participant 1 is sampled twice, their Mean RT in both samples is unlikely to be exactly equal. For each randomized/resampled dataset I recalculate d. In this case $N_{sim} = 10000$. What I'm observing is a trend for the observed value of Cohen's d to be typically closer to the 97.5th percentile of than to 2.5th percentile of simulated observed values. It also tends to be closer to 0 than the median of the bootstrap (by 5% to 10% of the density of the simulated distribution).

What can account for this (keeping in mind the magnitude of the effect I'm observing)? Is it due to it being 'easier' upon resampling to obtain more extreme variances than those observed relative to the extremity of means upon resampling? Might this be a reflection of data that has been overly massaged/selectively trimmed? Is this resampling approach the same as a bootstrap? If not, what else must be done to come up with a CI?

Answer 1

Any nonlinear statistic (a non-linear combination of linear statistics such as sample means) has a small sample bias. Cohen's $d$ is obviously no exception: it is essentially $$ d=\frac{m_1 - m_2}{\sqrt{m_3-m_4^2}} $$ which is fairly non-linear, at least as far as the terms in the denominator go. Each of the moments can be considered an unbiased estimator of what it's supposed to estimate: $$ \begin{array}{ll} m_1 & = \frac1{n_1} \sum_{i\in\mbox{group }1} y_i , \\ m_2 & = \frac1{n_2} \sum_{i\in\mbox{group }2} y_i , \\ m_3 & = \frac1{n_1+n_2} \sum_{i} y_i^2 , \\ m_4 & = \frac1{n_1+n_2} \sum_{i} y_i , \\ \end{array} $$ However, by Jensen's inequality there is no way on Earth you get an unbiased estimator of the population quantity out of a nonlinear combination. Thus ${\mathbb{E}}[ d]\neq$ population $d$ in finite samples, although the bias is typically of the order of $O(1/n)$. Wikipedia article on effect sizes mentions the small sample biases in discussion of Hedges' $g$.

I imagine that Cohen's $d$ has a limited range (in the extreme case, if there is no variability within groups, then $d$ must equal $\pm 2$, right?), hence its sampling distribution must be skewed, which contributes to the finite sample biases (some function of the skewness of the sampling distribution is typically the multiplier in front of $1/n$ that I mentioned above). The closer you are to the limits of the allowed range, the more pronounced the skewness is.

What bootstrap does, rather miraculously considering that it is such a simple method, is it gets you the ability to estimate this finite sample bias through comparison of the bootstrap mean and the estimate from the original sample. (Keep in mind though that unless you make special adjustments to how the bootstrap sampling is set up, the former will be subject to Monte Carlo variability.) I provided more detailed and more technical explanations in another bootstrap question which may be worth reading anyway.

Now if there's a positive bias, i.e., the estimate based on the original sample is biased upward relative to the population $d$, then the bootstrap will mock that and produce estimates that are, on average, even higher than the sample estimate. It is not actually as bad as it sounds, as then you can quantify the bias and subtract it from the original estimate. If the original estimate of a quantity was $\hat\theta_n$, and the mean bootstrap of the bootstrap replicates is $\bar\theta^*_n$, then the bias estimate is $\hat b_n=\bar\theta^*_n-\hat\theta_n$, and a bias-corrected estimate is $\hat\theta_n - \hat b_n=2\hat\theta_n - \bar\theta^*_n$.