1

I am creating the bootstrap to create a p-value.

Suppose my statistic from the sample is A. Suppose for simplicity, that I do 10 bootstraps and I get A_1, A_2, ..., A_10. Suppose that A_i > A two times and my alpha is .2. Should I reject or fail to reject?

I know that 10 bootstraps is nowhere near enough and that in practice I could just do more bootstraps until the p-value goes above or below. But I'm curious regarding this specific case.

Also how does this answer change if A_i = A for some i. Is that the same as A_i > A or does it count as A_i < A ? Again I don't see this case considered because A is often taken to be continuous. But what about non-continuous test statistic?

I've found this answer: Is a p-value of 0.04993 enough to reject null hypothesis? which suggests I hould reject if there is equality. But the particular case of equality is not explained. I have searched in elementary text books but most just say that p-value is continuous so equality won't happen.

I've also found this answer: Do you reject the null hypothesis when $p < \alpha$ or $p \leq \alpha$? but I don't have the book they mention and it is a little too theoretical for me. I guess my question is partly whether S_1 is {a| a > A} or {a | a >= A}?

I think the above questions have the material to answer my question but I don't know how to implement the knowledge with the specific case of a bootstrap.

My attempt at answer is that if A_i > A two times, we are saying Pr(A_i > A) = .2. We want probability of rejecting, given the null is true, to be .2. So we should reject. But as you can see I don't really understand things.

Community
  • 1
Xu Wang
  • 104
  • 14
  • 1
    Just as a general practical principle, when you are performing a bootstrap by means of simulation--or more generally when you are approximating any p-value--you should favor a *conservative* decision rule. This follows from the asymmetry in possible conclusions: "accepting" the null is more tentative than rejecting it. Rejection is an assertion that chance would be an inadequate explanation of the observed differences, whereas quite a bit of the distinction between $p\lt\alpha$ and $p\le\alpha$ is due to the operation of chance in your procedure. – whuber May 07 '14 at 18:58
  • I agree with whuber. Consider the case where $A_i > A$ 0 out of 10 times. What should be your estimated p-value? You can't really say that it should be 0, because you only did 10 random trials, right? Most likely, you would go with something like p < 1/10. (As a general rule, I report p < (k+1)/n for k "successes" in n random trials. So in your case, I would say p < 3/10, and you can't reject the null with $\alpha = 0.2$. – Hao Ye May 07 '14 at 21:12
  • If you don't reject when you have equality, then in the case of discrete test statistics, you simply don't attain the desired significance level under the null. (With truly continuous ones, exact equality is impossible, so it's a non-issue there.) The issue is perfectly clear when you work with predefined rejection regions, which I suggest you *always* do until the question is settled in your mind. – Glen_b May 07 '14 at 23:19
  • @HaoYe do you have a reference that discusses theory or other justification for your general rule? – Xu Wang May 08 '14 at 22:48
  • I think the "standard" p-value estimator is actually $p = (k+1)/(n+1)$ [following from my earlier notation], but $p < (k+1)/n$ naturally follows from that. (ref: http://dx.doi.org/10.1007/978-1-4757-3235-1) – Hao Ye May 08 '14 at 23:28

0 Answers0