Diagnostics for Bootstrap

Question

I would like to know if there are any agreed on diagnostics one should run after any simple bootstrapping procedure. For example, I bootstrap a quite simple parameter like a median or a correlation to get a standard error for these. The only thing I found so far is to check the distribution of the bootstrapped results to see whether they are normally distributed. Then I found some complex papers that were absolutely unintelligible to me and seemed to cover expert topics. I also found this paper here, which I still need to read: http://www.timhesterberg.net/bootstrap#TOC-What-Teachers-Should-Know-about-the-Bootstrap:-Resampling-in-the-Undergraduate-Statistics-Curriculum

Thanks for any advice.

Answer 1

Many novice statisticians mistakenly think you get a bootstrap sample and calculate a mean and SD to do significance testing and confidence limits. This is called the normal bootstrap. The only reason for its persistence is conceptual ease and computational efficiency.

In general the normal bootstrap shouldn't be used. The ways to calculate confidence intervals and significance tests with the bootstrap are: studentized, percentile, bias corrected accelerated (BCA), and double bootstrap. See a relevant post here: Computing p-value using bootstrap with R

If you use a correct bootstrap procedure, there's still no free lunch: some statistics still have invalid tests and summaries. At that point, the only way to know if the bootstrap method actually works is by performing a set of simulation studies. There's no way to inspect, without simulating data, whether a statistic is in fact pivotal, or if the test is of adequate power. This may have been why John Tukey famously declared "[The Bootstrap] should be called the shotgun because it blows the head off of any problem you have, provided you're willing to put the pieces back together."

Answer 2

When we take many re-samples for a nonparametric bootstrap, it is important to be mindful of how many uniquely different values of the bootstrapped quantity are actually generated. So I consider counting them to be an important diagnostic.

As an almost-trivial example, I illustrate with data x, which are an sample of size $n = 50$ from an exponential population with $\mu=10$ (expressed to 8 places). When I took $B = 10,000$ re-samples with replacement, I got $10,000$ uniquely different sample means, but only 136 uniquely different sample medians.

Details of the simulations in R are shown below:

# re-sampling medians
  set.seed(1234)
  x = rexp(50, .1);  h.obs = median(x)
  eta = qexp(.5, .1); eta
  [1] 6.931472
h.re = replicate( 10^4, median(sample(x, rep=T)) )
length(unique(h.re))
[1] 136

# re-sampling means
  set.seed(1234)
  a.obs = mean(x); a.obs = mean(x)
a.re = replicate( 10^4, mean(sample(x, rep=T)) )
length(unique(a.re))
[1] 10000