4

I've been thinking about what exactly a 95% confidence interval means when it is calculated via bootstrapping.

The formal definition of a 95% confidence interval is something like this: "if the population is repeatedly sampled and a confidence interval is calculated after each sample, then the population parameter being estimated will be included in the confidence interval 95% of the time".

But in bootstrapping, we don't calculate a confidence interval after each resample, we just calculate a single confidence interval at the end of the all of the resamples.

So the formal definition of a 95% confidence interval isn't transferable to a 95% confidence interval calculated via bootstrapping, because we don't calculate a confidence interval after each resample.

So, like Mike Lawrence asked, surely the 95% confidence interval calculated via bootstrapping must be interpreted as this: "there is a 95% probability that the confidence interval contains the population parameter we're trying to estimate". Yes?

luciano
  • 12,197
  • 30
  • 87
  • 119

1 Answers1

3

No, nothing has changed about the inferential situation, just the manner in which you generate the interval with the supposed properties.

So the interpretation of an CI generated this way doesn't change -- the big question is whether it really has the coverage properties claimed for it. That is, across many such samples, would the intervals generated in this manner include the parameter the specified fraction of the time?

That's a rather trickier question; often in small samples it seems that coverage can be rather smaller than the nominal coverage. Various things may improve the coverage in smaller samples in different circumstances, but I think that would be a discussion for another question.

Glen_b
  • 257,508
  • 32
  • 553
  • 939
  • I'm still struggling! If we resample 10,000 times, why don't we calculate 10,000 confidence intervals? – luciano Jun 29 '15 at 07:26
  • The point of bootstrapping a CI is to generate an interval without relying on a distributional assumption; (i) how are you going to get an interval from a single resample; and (ii) how will having 10000 intervals help you find out the coverage properties? – Glen_b Jun 29 '15 at 09:17
  • 1
    In the below link, this is how the definition of a confidence interval begins: "It means that if the same population is sampled on numerous occasions and interval estimates are made on each occasion…". This definition suggests to me that we should use the standard errors of each resampleto calculate a confidence interval and there is a 95% probability each confidence interval contains the population parameter. http://www.itl.nist.gov/div898/handbook/prc/section1/prc14.htm – luciano Jun 29 '15 at 09:40
  • Resampling from your *sample* is not getting a new sample of the population. – Glen_b Jun 29 '15 at 09:43
  • I thought that was the point of bootstrapping: to pretend the sample is the population and repeatedly take new samples from the population – luciano Jun 29 '15 at 09:47
  • 1
    With the bootstrap you do *not* estimate the interval in each replication but the point estimate. It is the distribution of these point-estimates that is an estimate of the sampling distribution of the point estimates. From that you derive the lower and upper bounds of your confidence interval, which has exactly the same interpretation as the confidence interval estimated by other means. – Maarten Buis Jun 29 '15 at 09:56
  • Let's come at this from another angle. How are you (luciano, not Maarten) going to generate a CI from each resample *without making some distributional assumption*? – Glen_b Jun 29 '15 at 10:00
  • 1
    Ok, so bootstrapping simply tells us what the sampling distribution of the point estimate would look like if we were to repeatedly sample. But despite trying hard, I can't see how this definition of a confidence interval is wrong; "there is a 95% probability the confidence interval contains the population parameter" – luciano Jun 29 '15 at 10:18
  • And now we've come back to my very first paragraph in my answer. How does using bootstrapping to obtain the interval make a difference to what the interval means? – Glen_b Jun 29 '15 at 10:45