6

I teach Introduction to Statistics and would like to show my students how to analyze the Pfizer Phase 3 Vaccine results. Testing a claim about 2 proportions is straightforward with the Pfizer data:

Given:

  • Group (1) is the placebo group;
  • Group (2) is the vaccine group

Data:

The number of people who got the 2nd dose of vaccine is 41,135 [A]. Assuming that people dropped out equally from the placebo and vaccine group, those groups were created at a 1:1 ratio [B]. So we have approximately 20,567 people in each group. Using the number of people who got covid from each group published in [A] we have:

Placebo Group: $x_1 = 162$, $n_1 = 20,567$

Vaccine Group (BNT162b2): $x_2 = 8$, $n_2 = 20,567$

Claim: $p_1 > p_2 \rightarrow p_1 - p_2 > 0$

$H_0: p_1 - p_2 \le 0$
$H_1: p1 - p2 > 0$

Test Statistic: $z = 11.8 \rightarrow \text{P-Value} < 0.0001$

Thus we have evidence for the claim that $p_1 > p_2$ meaning that we have evidence for the claim that the vaccine lowered the covid rate versus the control group.

So far, so good. But here's where things break down. The manufacturer's claim that the vaccine is shown to be 95% effective. If I use the 162 as the expected number of cases in general, then if the vaccine is 95% effective, the vaccine group should have less than 5% of the expected number of cases. 5% of 162 is 8.1 cases expected in 20567.

Control Group: $x_1 = 8.1, n_1 = 20,567$

Experimental Group: $x_2 = 8, n_2 = 20,567$

Claim: $p_1 > p_2 \rightarrow p_1 - p_2 > 0$

$H0: p_1 - p_2 \le 0$
$H1: p_1 - p_2 > 0$

Test Statistic: $0.0249 \rightarrow \text{P-Value} = 0.4901.$

This is clearly a fail meaning that with a P-Value of 0.4901, I have NOT shown that the vaccine group has less than 5% of the expected number of cases. Why is this analysis flawed? This seems related to (What does 94.5% effective mean?), but even reading this citation, its unclear to me why my hypothesis test is wrong. Again, an explanation appropriate for an INTRODUCTION to statistics class, please.

UPDATE: If I redo the analysis with an expectation of 16.2 cases instead of 8.1, then I get a p-value of 0.0477. This suggests that I can claim with a 5% significance level that the vaccine is 90% effective. Is the issue just that they can do a more detailed/sophisticated analysis to get to 95% effective or is my analysis just flawed and I've gotten lucky?

Christopher Donham
  • 171
  • 4
  • If I treat this as a confidence interval with x=8, n=162, CL=0.95, I get that I am 95% confident that the percentage of people who got COVID and had the vaccine is between 1.6% and 8.2%. Again, not so far off from the 5% published by Pfizer if all of this hangs together. Am I looking at this correctly? – Christopher Donham Dec 03 '20 at 13:57
  • 2
    I entered those values into an online [risk ratio calculation tool](https://www.medcalc.org/calc/relative_risk.php) and it gives a 95% confidence interval for effectiveness as 90% to 97.5%. That page has some details about the confidence interval calculation. It's a little complicated for an introductory class. – abstrusiosity Dec 03 '20 at 17:40
  • Can you help me to understand why my analysis failed? What makes the techniques I am using not-applicable to this situation? – Christopher Donham Dec 03 '20 at 22:32
  • What are you doing in this second computation? It is not clear how your computation relates to the hypothesis $H_0: p_1=p_2$ how did you compute the statistic 0.0249 and based on what distribution? You seem to be testing the hypothesis $H_0:p_2=8.1/20567$ via a normal approximation of a Poisson distribution? What do you mean that this test fails? – Sextus Empiricus Dec 09 '20 at 00:53
  • Aside from that, the test which is performed is not based on a z-test. What they do is condition on the number of cases (by performing the analysis once a certain number is reached) and treat the number of cases from placebo:vaccine groups as binomial distributed. (at least that is what Pfizer described in their protocol) of course you can make it more complex with survival analysis, Bayesian stuff, and computations that take into account age, gender, other background and computations based on time of vaccination and infection. See also https://stats.stackexchange.com/questions/496774 – Sextus Empiricus Dec 09 '20 at 00:59
  • Interesting. I was approximating binomial as normal which I believe is allowed if $(n)(p)(1-p) >= 10$ for both groups. In checking this assumption based on your comment, it appears to not hold up, indicating that normal distributions are not appropriate. – Christopher Donham Dec 09 '20 at 02:26
  • I also added some text to the question to try to better explain my thinking. – Christopher Donham Dec 09 '20 at 02:26

1 Answers1

1

Based on the comments, there are several issues with this analysis. First, the analysis is based on being able to approximate binomial as normal. Textbooks I've consulted write this requirement in different ways. Sullivan (Fundamentals of Statistics) says that in addition to being binomial, each of the groups must pass $np(1-p) \ge 10$ for this to be valid. For the control group this requirement is satisfied. For the experimental group it is not. Triola (Elementary Statistics) says the requirement is $np\ge5$ and $nq\ge 5$ for each of the two groups. This requirement is met for both groups.

Second, the second analysis also depends on the control group being an estimate of the number of cases expected in the experimental group. This assumption is inaccurate and could cause the vaccine to appear less effective than it is.

Finally, for the confidence interval, again the textbooks give differing requirements ($np(1-p) \ge 10$ vs $np \ge 5$ and $nq \ge 5$). Only some of these requirements are met, suggesting we are on the hairy edge of validity with this analysis. So a crude guess about the proportion who got covid is ($1.6\%$ to $8.2\%$).

What all of the means to me is that we can do some basic analysis, but this has to be taken with a grain of salt because the number of cases is still sufficiently small to not robustly meet the requirements.

Finally, thanks to the people who commented on this posting as they helped me to think more deeply about what was going on

Christopher Donham
  • 171
  • 4