12

This question is in regard to AstraZeneca's phase 3 clinical trial on the effectiveness of their COVID-19 vaccine.

Patients were randomized to a two dose treatment group compared to a two dose placebo group. But a large portion of the treatment group inadvertently received a lower first dose. I understand that the group getting a low dose first consists of 2,741 subjects & the full dose group contained 8,895 subjects.

Since the result that those taking the lower dose first and the full dose later was statistically significantly more effective (at approximately 90% compared to 62%) than the group that got the full dose both times, shouldn't we discount it?

Those getting the low dose first got it by accident. So they don't represent a randomized group & therefore isn't it statistically unsound to compare them to the control group & the other treatment group? Also, I would think that the estimates of 90% for the low dose first group, the 62% for the 2 high dose group & the overall 70% effectiveness for the combined treatment groups be considered invalid because of possible bias?

Dave
  • 28,473
  • 4
  • 52
  • 104
Michael R. Chernick
  • 39,640
  • 28
  • 74
  • 143
  • 3
    Which results, specifically? For the record, and to keep this question self-contained, what was the nature of the "accident" and why would it not be representative? – whuber Nov 28 '20 at 15:55
  • I was referring to the estimate that the vaccine is statistically significantly more effective in the group getting the lower dose compared to the other treatment group. – Michael R. Chernick Nov 28 '20 at 16:14
  • 3
    Right: but to make this question self-contained, please supply the essential information about the study to which you are referring. – whuber Nov 28 '20 at 16:20
  • I've added a more detailed description of the trial. Because the information has thus far only been given in a press release & we don't have complete information on the number of patients in each group yet, this is all the detail I can give. But I think it is enough to answer the question. – Michael R. Chernick Nov 28 '20 at 16:40
  • 4
    Thank you. I have reopened it, but hope you might (either now or eventually) be able to supply the actual counts, because those are essential for determining whether we should even consider 90% to differ significantly from 62%. – whuber Nov 28 '20 at 16:42
  • @whuber I agree of course that the group sizes are necessary to determine statistical significance in the difference between the 2 treatment groups if the trial was truly a randomized controlled trial. But the issue is whether inference can be valid when there is a lack of randomization. – Michael R. Chernick Nov 28 '20 at 16:51
  • To clarify, are there 2 main questions? 1) i.i.d. assumption violated: since the subgroup (the lower dose) was not identically distributed to the rest of the treatment group due to the lower dose; 2) double-test problem: since a subgroup of the treatment group (the one with lower dose) got significantly better results than the rest (test 1), would this skew the results of comparing only this subgroup to the control (test 2)? – PaulG Nov 28 '20 at 17:08
  • On further review I learned the number of subjects in the treatment groups are 2,741 in the low dose first group & 8,895 in the group that got the full dose in both treatments. I do not yet have information on the number of subjects in the control group, My question is whether or not we can draw inferences about the difference in effectiveness between the two treatment groups since we cannot assume that they are random samples from the population of subjects. – Michael R. Chernick Nov 28 '20 at 17:32
  • 1
    @Michael R. Chernick A further wrinkle. According to an article in Science Alert, all of the 2,741 people who received the low dose first were in the UK. https://www.sciencealert.com/scientists-are-trying-to-understand-how-effective-the-oxford-vaccine-really-is – Diana Petitti Nov 28 '20 at 18:21
  • @DianaPetitti As you point out since all 2741 all come from the UK ethnicity becomes an issue. The UK has a higher percentage of whites compared to Brazil & South Africa. – Michael R. Chernick Nov 28 '20 at 18:38
  • 2
    @Michael R. Chernick This article in New Scientist provides even more detail about how the error occurred and clarifies the fact that there were really two studies going on with different protocols--one in the UK and one in Brazil. In addition to the mistake in dose, there were other differences in the two protocols. https://www.newscientist.com/article/2261092-do-oxford-astrazeneca-covid-19-vaccine-results-stand-up-to-scrutiny/ – Diana Petitti Nov 28 '20 at 18:45
  • So the New Scientist article also shows an age distribution issue. The UK subjects in the low dose first had a higher distribution of young patients than other groups & younger subjects are expected to fair better against COVID. – Michael R. Chernick Nov 28 '20 at 19:00

1 Answers1

4

My opinion is no. I felt that AZ's primary publication was high quality: reporting the planned analyses, presenting a very important unplanned analysis, explaining the unexpected results of that analysis, and claiming that more time and data will be needed to confirm any findings.

The question boils down to intent-to-treat versus per-protocol analysis. Every trial has unexplained deviations, and sometimes it even affects manufacturing so that a large fraction of trial participants are affected by a non-randomized condition. This can be so severe that the planned analyses need to be reframed to report any generalizable result. Unfortunately for AZ who seem to have a fairly good vaccine, all of the above happened with their ChAdOx trial. This cost them vital time and "vitaler" money in a highly competitive environment.

Deviations and unexpected findings can at times generate hypotheses that are so striking they need to be evaluated in subsequent studies. But the operative word is "subsequent": you can't use the same data that generated a hypothesis to confirm the hypothesis. Minoxidil, for instance, was a vasodilator piloted as a heart medication but unintentionally was found to cause hair grown in those with male pattern baldness. They conducted a secondary study and showed a statistically significant effect. For AZ, if they truly believe the low dose or "priming low dose" was effective, they need a major protocol amendment, and an analysis plan that excludes results from the prior analyses; a very costly endeavor. I recall this was investigated at length, the priming dose hypothesis was ultimately rejected. Now ChAdOx has emergency use authorization from WHO and is being used globally (outside the US).

In general, when reporting out the results of a clinical trial, it is better to verge on the side of full disclosure. According to my reading of the primary publication, they report correctly the intent-to-treat (so called "average" effectiveness) of 70%. The "per protocol" effectiveness of the AZ drug was also reported (among those who received both full doses) as 62·1% (95% CI 41·0–75.4) which is still a promising result. They correctly report the results of a rather serious post-hoc finding: that people receiving an "accidental" dose had statistically better efficacy that those receiving the planned dose.

In participants who received two standard doses, vaccine efficacy was 62·1% (95% CI 41·0–75·7; 27 [0·6%] of 4440 in the ChAdOx1 nCoV-19 group vs71 [1·6%] of 4455 in the control group) and in participants who received a low dose followed by a standard dose, efficacy was 90·0% (67·4–97·0; three [0·2%] of 1367 vs 30 [2·2%] of 1374; pinteraction=0·010). Overall vaccine efficacy across both groups was 70·4% (95·8% CI 54·8–80·6; 30 [0·5%] of 5807 vs 101 [1·7%] of 5829).

The article goes farther to state that the results are unexpected, that they performed other subgroup analysis to see if possible confounding could explain the issue, and no statistical test identified a difference. They clarify further that the results, while promising, will require later read out of the data and results from other studies to understand any difference if not due to chance.

The issue had to do with manufacturing. The primary publication is open access and you can read it for yourself: https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)32661-1/fulltext#tbl2 (open access). The low dosing issue was one of lack of oversight from AZ.

Two dosage groups were included in COV002: participants who received a low dose of the vaccine (2·2 × 1010 viral particles) as their first dose and were boosted with a standard dose (in the LD/SD group), and subsequent cohorts who were vaccinated with two standard-dose vaccines (SD/SD group). Initial dosing in COV002 was with a batch manufactured at a contract manufacturing organisation using chromatographic purification. During quality control of this second batch, differences were observed between the quantification methods (spectrophotometry and quantitative PCR [qPCR]) prioritised by different manufacturing sites. In consultation with the national regulator (Medicines and Healthcare products Regulatory Agency), we selected a dose of 5 × 1010 viral particles by spectrophotometer (2·2 × 1010 viral particles by qPCR), in order to be consistent with the use of spectrophotometry in the phase 1 study (COV001),5 and to ensure the dose was within a safe and immunogenic range according to measurements by both methods. A lower-than-anticipated reactogenicity profile was noted in the trial, and unexpected interference of an excipient with the spectrophotometry assay was identified. After review and approval by the regulator, it was concluded that the qPCR (low-dose) reading was more accurate and further doses were adjusted to the standard dose (5 × 1010 viral particles) using a qPCR assay. The protocol was amended on June 5, 2020, resulting in enrolment of two distinct groups with different dosing regimens with no pause in enrolment (version 6.0; appendix 2 p 330). A suite of assays has now been developed for characterisation of concentration (which confirmed the low and standard dosing), and future batches are all released with a specification dose of 3·5–6·5 × 1010 viral particles, and this was used for the booster doses in the efficacy analysis presented here.

AdamO
  • 52,330
  • 5
  • 104
  • 209