no repetition confidence intervals

Question

Say 2% of the population dies before a drug is consumed and 1% of the population dies after the drug is consumed. Then, a non-mathematician will say yes the drug was effective. What better can we say instead?

How should I design this same experiment in a better way to say something statistically? My total population can be assumed to remain the same (say it is very large, so despite 2% dying it doesn't change much). But I cannot repeat this experiment so I cannot design a $t$ test around it. I don't even know how I can develop a CI if I cannot repeat this experiment.

Answer 1

Say 2% of the population dies before a drug is consumed and 1% of the population dies after the drug is consumed. Then, a non-mathematician will say yes the drug was effective. What better can we say instead?

If you have the entire population, then you're good. You for sure that the percentage of people dying was cut in half after the drug was introduced.

But, let's say you don't have the population. Let's say you have 1000 people with Condition X before a drug is introduced. Two percent of these unfortunately pass away. Then a drug is introduced, and you look at 1000 more people after this. Only 1% of this sample passes away.

You could analyze this, for example, with a logistic regression. Each person would be their own row in the data, there would be a variable indicating if they got the condition before or after the drug was introduced. Your regression would look like y ~ drug where y indicates if they passed away (1) or not (0).

However, this would not be a causal interpretation. It could have been that a strain of the bacteria Condition X got weaker as time went on, so it wasn't due to the medicine at all.

This gets to the next question.

How should I design this same experiment in a better way to say something statistically? My total population can be assumed to remain the same (say it is very large, so despite 2% dying it doesn't change much). But I cannot repeat this experiment so I cannot design a test around it. I don't even know how I can develop a CI if I cannot repeat this experiment.

Again, if you have the entire population, then you're good. You don't need statistics, since statistics are just used to infer from a sample to a population.

But what I would suggest is taking a sample of people with some Condition X, randomly assign them to either get a drug or a placebo, and then test the rate at which patients passed away with the same regression model as described above. However, now, due to random assignment, you could make a causal claim.

On "repetition"

I want to address the idea of "cannot repeat this experiment." Frequentist statistics, which confidence intervals come from, pose a pretty theoretical question when it comes to these confidence intervals: "Imagine I ran this exact same experiment an arbitrarily large amount of times. What would the death rate be in 95% of those experiments?" But we only have one sample. So we rely on assumptions in our models and methods that lets us make these long-run, frequentist interpretations. So you don't need to repeat the experiment a large number of times. You can rely on statistics to get an idea of uncertainty. And you can decide if you're willing to tolerate that level of uncertainty and call the 1% meaningfully different from the 2%.

Answer 2

I don't understand why you say you can't repeat the experiment. The way a drug trial is normally run, you divide study participants into two groups. In an idealized and very simple example, group 1 has $N_{1}$ participants, and they are treated with the drug, while group 2 has $N_{2}$ participants, and they are treated with a placebo. Each time a patient is treated with either the drug or a placebo then constitutes an "experiment", which is repeated $N_{1}$ times for group one, and $N_{2}$ times for group 2. After an appropriate length of time, a proportion $p_{1} = n_{1} / N_{1}$ of the participants in group 1 has died, and $p_{2} = n_{2} / N_{2}$ has died. We can think of these proportions, $p_{1}$ and $p_{2}$, as probabilities in a binomial probability distribution. Effectively, we may imagine that each patient flips a biased coin (roughly 98% vs. 2%) to determine whether he lives or dies; the question is, does the drug improve the coin bias in the patient's favor, or not? Within this simple model, we can define a null hypothesis and an alternative hypothesis:

• $H_{0}$: $p_{1} = p_{2}$ (null hypothesis; drug has no effect, so the two underlying binomial distributions are the same)

• $H_{1}$: $p_{1} \neq p_{2}$ {alternative hypothesis: drug has some effect, usually to improve outcomes, $p_{1} < p_{2}$, but in principle it could also make things worse, $p_{1} > p_{2}$)

This particular pair of hypotheses has a t-test already defined in the literature; it's typically called the t-test of binomial proportion, or some similar name. Example discussions are availble here, here, here, here and here. Additionally, this source attempts to provide some theoretical derivation.

Caveat: in real life, drug trials often have complications that make a simple statistical model such as this somewhat unsuitable:

• Patients in real studies are not usually enrolled at equal times, but often over a period of up to a few years, with the result that those who were enrolled at the beginning of the study may have been tracked for years, while those enrolled toward the end were only tracked for a few months. Obviously, a patient's probability of dying increases the longer the study tracks him.

• Some fraction of patients will inevitably leave the study early (for example, they move away to another city to find another job) before the study is finished. In these cases we know that they survived up to a certain date, but then we don't know what happened after that.

• For some drugs, such as many cancer drugs nowadays, or antiretrovirals for HIV/AIDS treatment back in the 1990s, the relevant question to ask is not whether a patient lived or died in response to the drug, but did the patient live longer than he otherwise would have (e.g., did he survive a year instead of dying after only 3 months)?

Kaplan-Meier survival curves are a typical framework for analyzing these types of situations.