3

Can anyone give an explanation of the data generating process for the non-parametric bootstrap?

By data generating I refer to randomly sampling from the population (if it is known) and then resampling with replacement from the sample.

What sample size do we use? Bootstrap is often used when the sample size is fairly small, lets say n < 15, but it fails mostly when n < 5.

Thanks a lot!

Stochastic
  • 799
  • 1
  • 6
  • 28
  • 1
    It's not quite clear what you mean by "data generating process", and there are various versions of bootstrap, notably parametric, nonparametric, and several variants for specific situations. – Christian Hennig Jun 06 '20 at 21:38
  • Apologies @Lewian I realised that my question was a bit unclear. I am familiar with bootstrap, but looking for a more clear explanation – Stochastic Jun 07 '20 at 09:44
  • 3
    The standard is to use the same sample size for bootstrapping that is the sample size of the data (although there are variants, see. e.g., https://stats.stackexchange.com/questions/96739/what-is-the-632-rule-in-bootstrapping ). If the sample size is very small, bootstrap can't generate that much variation, so it won't work that well, but then hardly anything nonparametric will work better. As always there is no fixed rule about sample sizes... bootstrap is worse with a smaller sample, but all alternative approaches are worse, too, so it is not clear cut. – Christian Hennig Jun 07 '20 at 12:37
  • Thanks a lot @Lewian that clears it up – Stochastic Jun 07 '20 at 12:42
  • If this is what you wanted to know, I can make it an answer. – Christian Hennig Jun 07 '20 at 12:48
  • It answers the second part, not the first part entirely, but I will accept the answer, thanks @Lewian – Stochastic Jun 07 '20 at 15:19

3 Answers3

5

You seem to be asking two different questions, the main one about data generating process for bootstrap, and the second one about sample size. Your second question seem to have got answered, but not the main one.

Data generating process is "the processes that cause data to occur as they do", so about the unobserved probability distribution that "generated" your data $\mathbf{x} \sim P$. Bootstrap is not concerned about the underlying probability model $P$, but only about the uncertainty of the estimates $\hat\theta$ due to sampling. Bootstrap works by imitating the process of taking the sample $\mathbf{x}$ from the population $P$, with sampling randomly with replacement from the data (i.e. from the empirical distribution $\hat P$). This is illustrated on the figure from Second Thoughts on Bootstrap by Efron (2003).

enter image description here

Tim
  • 108,699
  • 20
  • 212
  • 390
  • Thank you so much, this is extremely helpful and very well explained. I understand the basic concepts of the bootstrap, but it has been difficult for me to explain it to other people. – Stochastic Jun 08 '20 at 07:43
3

There are many styles of nonparametric bootstrap. For a random sample of size $n$ from a population with mean $\mu$ the principle of the bootstrap is to regard the empirical CDF of the data as an estimate of the population CDF.

One takes a large number $B$ of re-samples of size $n$ from the data x with replacement and finds the mean of each re-sample. A very simple bootstrap takes quantiles .025 and .975 of these re-sampled means as a 95% confidence interval for $\mu.$

All we know about the population is that it is capable of producing the $n=15$ observations in our sample, so we re-sample from them. [Re-sampling without replacement would make no sense, because a re-sample without replacement gives just the same sample again--likely re-arranged, but with exactly the same $n$ values.]

Example: Data in vector x in R.

summary(x);  sd(x);  length(x)
       Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
     0.1139  3.4503  7.3113  6.1035  7.8384 11.0812 
[1] 3.189805
[1] 15
stripchart(x)

enter image description here

Because the sample shows no distinct skewness, it is OK to use a simple quantile bootstrap procedure without bias correction as shown below, obtaining the 95% nonparametric bootstrap CI $(4.46, 7.60).$

set.seed(2020)
B = 3000;  n = length(x)
a.re = replicate(B, mean(sample(x, n, rep=T)))
CI = quantile(a.re, c(.025,.975));  CI
    2.5%    97.5% 
4.459446 7.597443

Here is a histogram of the $B = 3000$ re-sampled averages, with vertical red lines indicating the quantiles used to make the bootstrap.

hist(a.re, prob=T, col="skyblue2")
 abline(v = CI, col="red")

enter image description here

By contrast, here is a parametric 95% t confidence interval $(4.34, 7.87),$ which assumes that the data are normal.

t.test(x)$conf.int
[1] 4.336995 7.869906
attr(,"conf.level")
[1] 0.95

A direct comparison of the two CIs is unwarranted. The bootstrap CI assumes only that the data were randomly sampled from some population with mean $\mu.$ The t CI assumes random sampling from a normal population.

Note: In this case, $n=15$ observations were randomly sampled from $\mathsf{Unif}(0,12)$ so that $\mu = 6.$

set.seed(1234)
x = runif(15, 0,12)
BruceET
  • 47,896
  • 2
  • 28
  • 76
2

Not quite sure whether this addresses the first part of the question, but regarding sample sizes:

The standard is to use the same sample size for bootstrapping that is the sample size of the data (although there are variants, see, e.g., What is the .632+ rule in bootstrapping? ). If the sample size is very small, bootstrap can't generate that much variation, so it won't work that well, but then hardly anything nonparametric will work better. As always there is no fixed rule about sample sizes... bootstrap is worse with a smaller sample, but all alternative approaches are worse, too, so it is not clear cut.

Christian Hennig
  • 10,796
  • 8
  • 35