Why is bootstrapping p-values not just finding the null in the bootstrap samples?

Question

I've been trying to look into the possibility of plucking a p-value for a slightly tricky case from the bootstrap distribution that I'm generating to construct confidence intervals. Everything I'm reading, including here on StackExchange (e.g. https://stats.stackexchange.com/a/277391/54668) and elsewhere, talks about rejigging the data so that the bootstrap samples represent the distribution of the statistic under the null. This makes sense as a way forward. But what I don't understand is, why can we not just look at the percentile of the null hypothesis parameter value in the bootstrap samples we used to generate our CI? I know that is not what the bootstrap samples are supposed to model, but my thinking is as follows:

I understand that the sampling distribution (if the alternative is true) can be wildly different from that if the null is true and further that the bootstrap samples model the sampling distribution of the estimate... But... the following logic then leaves me confused. For simplicity of explanation, I'll talk about one-sided CIs and tests ($H_0: \mu=0, H_1: \mu > 0$). And I'll stick to percentile CIs for simplicity (suppose we can assume they will be good in this case):

Suppose the null ($\mu=0$) is true. For a $(1-x)\%$ CI, it will miss zero, $x$% of the time, giving $x\%$ falsely significant results (at the $x\%$ level).

Take a tiny $\epsilon$. If the $(x+\epsilon)$th bootstrap percentile lies just above our null (0), then our null (0) lies outside the CI, it is a false positive and statistically significant at the $(x+\epsilon)\%$ level, and $p<(x+\epsilon)$. Conversely, if the $(x-\epsilon)$th percentile lies just below zero, then $p>(x-\epsilon)$. So surely, for that $x$, $(x-\epsilon)<p=x<(x+\epsilon)$? I.e. $p=x$ is the percentile of the null in the bootstrap samples. Is there a gap in this logic? Or is there another reason why we need to do all this shifting around of data to recreate the null distribution?

Answer 1

Hi: I think what you're missing is that, for bootstrapping to work, not only does the distribution of the "thing" being bootstrapped have to converge to a distibution under the null but that "thing" has to be pivotal. By pivotal, it is meant that the statistic being bootstrapped doesn't depend on the parameter being tested under the null.

But, if we use the bootstrapped samples themselves, then, clearly, that's not true. If we generate samples from the original population, then the bootstrapped distribution of the sample clearly depends on the value of $\mu$.

The idea of bootstrapping is to be able to avoid a distributional assumption about the original sample by using the fact that the constructed pivotal statistic from the sample ( hopefully) converges to a distribution. This way, we can look at the resulting distribution of the pivotal statistic and see where the actual statistic from the original sample falls within that distribution. I hope that helps.