Computing p-value in a problem about people going to a show with certain probability

Question

I'm having trouble understanding this question from my statistics class:

I handed out a flyer about a rock show to 20 people who, at that time, did not plan on attending the show. My hypothesis is that 5 or more people are now going to the rock show. I decide to follow up with 8 of them (chosen randomly) to see if they went to the show. I find that 2 of those 8 people went to the show. What is the p-value of the outcome of my experiment?

What type of statistical test is necessary to find the p-value?

Answer 1

This question has a unique answer: anything else will only be an approximation or will be based on an inferior hypothesis test. The p-value is $682/969 \approx 70.4\%$. Its calculation is based on sampling without replacement from a population of $20$. The rest of this post provides the reasoning, which relies only on the definition of p-value and some straightforward combinatorial calculations.

---

A probability model for your results can be described by a box with $20$ tickets, one per person. On each ticket is written whether that person went to the show. Your random sample is like taking eight of those tickets out of the box (without replacing them).

The statistical characteristics of this model which are yet unknown are completely determined by the number of people who went to this show. Call this number $\theta$. The possible values of $\theta$ are the whole numbers from zero through $20$.

Your null hypothesis, $H_0$, is that five or more people went to the show: $\theta \ge 5$. The alternative is that $\theta \lt 5$.

To test this hypothesis, the only useful statistic is the count $X$ of the people in your sample who went to the show. (Counting those who did not go will give mathematically equivalent information, obviously.) Evidently small values of $X$ are evidence against $H_0$ and large values are evidence for it. In fact, if $X \ge 5$, you would be certain that $H_0$ is true, because at least five people in your sample went.

The p-value therefore is computed from the chance that $X$ could have been less than or equal to the value you observed, which was $2$. This chance can easily be computed by breaking it into three mutually exclusive possibilities:

• $X=0$ means the entire sample of $8$ tickets came from the $20-\theta$ non-show-going tickets in the box. There are $\binom{20-\theta}{8}$ ways that could happen.

• $X=1$ means seven of the sample tickets came from the $20-\theta$ non-show-going tickets (there are $\binom{20-\theta}{7}$ ways for that to happen) and one came from the $\theta$ show-going tickets: there are $\binom{\theta}{1}$ ways for that to happen, independently of the choice of the other seven tickets. The total number of such samples therefore is $\binom{20-\theta}{7}\binom{\theta}{1}$.

• An analogous argument shows there are $\binom{20-\theta}{6}\binom{\theta}{2}$ samples with $X=2$ show-going tickets.

Add these three values up and divide by the total number of possible (and equiprobable) samples, $\binom{20}{8}$, to obtain the chances of $X\le 2$ in terms of the unknown $\theta$. Although as it turns out we only need to perform this calculation for $\theta=5$, here are the chances for some of the other values of $\theta$ so you can appreciate the patterns:

$$ \begin{array}{rr|cccccc} &\theta & 2 & 3 & 4 & \color{Red} 5 & \color{Red} 6 & \color{Red} \cdots & \color{Red} {14} \\ &\text{Probability} & 1 & \frac{271}{285} & \frac{4103}{4845} & \color{Red}{\frac{682}{969}} & \color{Red}{\frac{176}{323}} & \color{Red}\cdots & \color{Red}{\frac{7}{9690}} \\ &\text{(in decimals)} & 1. & 0.951 & 0.847 & \color{Red} {0.704} & \color{Red} {0.545} & \color{Red}\cdots & \color{Red} {0.001}\\ \end{array} $$

(I started the table at $\theta=2$ because you already observed two show-goers in your sample. I ended it at $\theta=14$ because you already observed six non-show-goers, leaving at most $20-6=14$ show-goers.)

When $\theta$ is small (which it is under the alternative hypothesis, consisting only of the possibilities $\{0,1,2,3,4\}$), the chance that $X\le 2$ is high. But as $\theta$ increases, the chance goes down. Among the null hypothesis, which comprises the cases $\theta=5, 6, \ldots, 20$ (tabulated in red), the greatest chance occurs when $\theta=5$, where it is $682/969 \approx 0.704$. This is the p-value.

Let's interpret this conclusion to check that it makes sense. The narrative might go like this:

I wish to test whether there are five or more show-going tickets in the box. A small number of show-going tickets in my sample would be evidence against that. I saw just two show-going tickets in the sample. There actually is a situation--namely, where exactly five out of the twenty people went to the show--where the chance of observing two or fewer tickets in my sample is as great as $70.4\%$. This is very high, showing my sample is consistent with the null hypothesis.

---

As a further check of this reasoning, consider a scenario in which your sample ought to have a low p-value. Suppose your null hypothesis were that more than half of the 20 people went to the show. The corresponding set of possible values of $\theta$ is $H_0 = \{11, 12, \ldots, 20\}$. The largest chance that $X\le 2$ for any of those situations occurs when $\theta=11$ and is only $4\%$ ($335/8398$). That's a pretty low p-value, allowing you to conclude it's likely there are fewer than $11$ show-going tickets in the box. Indeed, you have seen only two of them and there are just $12$ tickets left in the box, so you would be confident there are fewer than nine show-going tickets among them. Given that only one-quarter of the sample contains show-going tickets, that's a reasonable conclusion.

Answer 2

I think this homework question is a bit unclear.

The experiment seems to be handing out the flyer to a population of $20$ people. You want to know what is the probability that $5$ to $20$ people attend as a result. Usually, experiments involve a control group that gets a placebo, but perhaps this is a new band that no one knows about, so the assumption that no one plans to attend in the absence of the treatment flyer is reasonable. Or maybe the organizers really liked this study. The second assumption is that the flyers cannot be shared, so treatment is not "contagious". This is less reasonable, but would complicate the problem too much to relax. In any case, if you think of an experiment as a procedure carried out to verify, refute, or validate a hypothesis, this fits the bill.

The problem is that you don't know the take-up rate in the population. To learn it, you sample $8$ receivers at random from the population of $20$, and note that $2$ went. The rate seems to be $1/4$. You might even do a binomial test here and find that you cannot reject the null that $p=0.25$, as in @E L M's answer.

The ultimate goal, however, is to to extrapolate from your sample to the population of 20, which is the experiment. The probability that $5$ or more people attend when you hand out $20$ flyers can be calculated by the binomial tail function, which gives your the probability of observing $k=5$ or more successes in $20$ trials when the probability of a success on one trial is $p=\frac{1}{4}$. In Stata, this would be:

. display binomialtail(20,5,1/4)
.5851585

You can even do this from first principles by subtracting $1 - Pr(k=0)-Pr(k=1)-Pr(k=2)-Pr(k=3)-Pr(k=4)$:

 di 1-[binomialp(20,0,1/4)+binomialp(20,1,1/4)+binomialp(20,2,1/4)+binomialp(20,3,1/4)+binomialp(20,4,1/4)]
.5851585

You could also think of this as a one-sided binomial probability test:

. bitesti 20 5 1/4

        N   Observed k   Expected k   Assumed p   Observed p
------------------------------------------------------------
       20          5            5       0.25000      0.25000

  Pr(k >= 5)           = 0.585158  (one-sided test)
  Pr(k <= 5)           = 0.617173  (one-sided test)
  Pr(k <= 5 or k >= 6) = 1.000000  (two-sided test)

The first one-sided test gives you the same probability as the tail approach. It is also a p-value. Why?

The p-value of a hypothesis test is the probability (calculated assuming $H_0$ is true) of observing any outcome as extreme or more extreme than the observed outcome $(k=5)$, with extreme meaning in the direction of the alternative hypothesis. You reject the null when the p-value is small, in favor of the alternative, because anything as extreme or more is unlikely if the null was true. You don't accept the null, however, the data can only be consistent with it.

In R, this can be done with:

> binom.test(5,20,1/4, alternative = "greater")

        Exact binomial test

data:  5 and 20
number of successes = 5, number of trials = 20, p-value = 0.5852
alternative hypothesis: true probability of success is greater than 0.25
95 percent confidence interval:
 0.1040808 1.0000000
sample estimates:
probability of success 
                  0.25