1

I'm trying to understand how a frequentist would approach statements like "having observed X, the probability of Y happening is Z".

For concreteness, let's say we conducted $n_1$ identical, independent Bernoulli trials, and observed $n_1$ successes and 0 failures. We want to attach a probability to the statement "the next $n_2$ trials will also all succeed".

The Bayesian approach is straightforward: choose a prior for the Bernoulli parameter $p$, let's say a Beta distribution $\mathrm{Beta}(\alpha, \beta)$, incorporate the likelihood of the first $n_1$ observations to obtain a posterior, then compute the expected value of the probability of $n_2$ additional successes according to the posterior. If I'm not mistaken, the result is:

$$\text{Prob}(\text{$n_2$ additional successes}) = \frac{B(\alpha+n_1+n_2, \beta)}{B(\alpha+n_1,\beta)}$$

I'm less sure about how a frequentist would look at the problem. Here is what I think. We decide a priori that we are going to make our claims according to the following decision tree:

  • If there is any failure in $n_1$ trials, then make no claim.

  • If there are no failures in $n_1$ trials, then claim that there will be no failures in the next $n_2$ trials.

The probability that this decision tree will lead to make an incorrect claim is:

$$\text{Prob}(\text{incorrect claim}) = p^{n_1}(1-p^{n_2})$$

and this probability is maximized if the Bernoulli parameter is $$p_{\text{worst case}} = \left( \frac{n_1}{n_1+n_2}\right)^{\frac{1}{n_2}}$$

So the probability of making an incorrect claim, in the worst case scenario, is: $$\text{Prob}(\text{incorrect claim, worst case}) = \frac{n_2}{n_1+n_2}\left(\frac{n_1}{n_1+n_2}\right)^{\frac{n_1}{n_2}}$$

And this would lead me to conclude that, from a frequentist point of view:

$$\text{Prob}(\text{$n_2$ additional successes}) = 1 - \frac{n_2}{n_1+n_2}\left(\frac{n_1}{n_1+n_2}\right)^{\frac{n_1}{n_2}}$$

Here the meaning of the probability is that if the experiment is repeated many times, for every time that the decision tree leads to claim there will be $n_2$ additional successes, the claim will wrong at most this fraction of the time, regardless of the true value of $p$.

I am suspicious of this approach mainly because it appears to be less conservative than the Bayesian, for any reasonable choice of the prior.

Andrea Allais
  • 111
  • 1
  • 1
    Does https://stats.stackexchange.com/questions/26702 answer your questions? How about https://stats.stackexchange.com/questions/17773? – whuber Nov 18 '20 at 19:17
  • 1
    Thanks @whuber 26702 seems very relevant. Let me see if I am able to reformulate this problem in that language. – Andrea Allais Nov 18 '20 at 19:43
  • 2
    Thanks @whuber, I now understand that the frequentist approach is to use prediction intervals, a concept that sadly I did not know about. I also found useful https://stats.stackexchange.com/questions/255570 and references therein. – Andrea Allais Nov 18 '20 at 21:25
  • Good find, thank you! – whuber Nov 18 '20 at 21:28
  • Does this answer your question? [Prediction and Tolerance Intervals](https://stats.stackexchange.com/questions/26702/prediction-and-tolerance-intervals) – Peter O. Nov 19 '20 at 04:40

0 Answers0