Understanding the "multiple comparisons problem"

Question

I am trying to understand the multiple comparisons problem based on the wikipedia article below:

https://en.wikipedia.org/wiki/Multiple_comparisons_problem

In this article they take the following example:

Suppose we consider the efficacy of a drug in terms of the reduction of any one of a number of disease symptoms. As more symptoms are considered, it becomes increasingly likely that the drug will appear to be an improvement over existing drugs in terms of at least one symptom.

I don't understand why this is a problem. Because in my opinion, if I am testing a new drug to solve a disease and that disease produces a syndrome (i.e. a set of symptoms) then knowing that it may solve one symptom (with a x% confidence) is not a passed test for me.

In my opinion if I test the efficacy of the drug on n symptoms, when n increases I reduce the chance of being misled on the efficacy of the drug. Indeed, if I only test the effect of the drug on one symptom then if by misfortune the test has an error I could say that the drug is efficient.

Can someone explain to me what am I not understanding here?

Answer 1

If you have a test significant at $p=0.1$ level, this means that given that null hypothesis is true, there is $0.1$ probability that the result comes from the null distribution. $10\%$ of the time you would be rejecting null hypothesis nonetheless it is true. Saying it differently, there is $0.9$ probability that if null hypothesis is true, the results do not come from the null distribution. Say that you run ten such tests, in such a case, the probability that you correctly reject null hypothesis in all cases is

$$ \underbrace{0.9\, \times\, 0.9 \, \times\, ... \,\times\, 0.9}_{10 \times} \approx 0.35 $$

and the probability that at least one of the tests to be falsely rejected is

$$ 1 - 0.9^{10} \approx 0.65 $$

So if you test more symptoms, you are more likely to conclude that the drug works for each new symptom even if it doesn't. In such a case, not all of the results that you observed are plausible. When you test a single symptom, you are much more specific and have less risk of falsely rejecting the null hypothesis.

Answer 2

Maybe the phrasing of testing a new set of symptoms is throwing you. Imagine you tested the same set of symptoms again and again. Eventually, the test distribution under the null hypothesis, even if it truly is null in the world, will randomly show a significant (improbable) result. That's the idea. Keep testing and eventually you'll get a positive result, doesn't mean it's a true positive.

But it still applies if you test different symptoms. Everytime you looked you've used the same null hypothesis--that the test statistic will be less than whatever critical value gives p=0.05. Eventually, any distribution will generate improbable values.

Imagine you have some chemical compound and you say "Well it's got to do something to the human body! Keep looking!" Eventually you'll find a set of symptoms that will ping significant.

If the idea of syndromes is a hangup, people often condense multidimensional ideas into one single measure to be able to test it. Think making a psychological measure of depression, which often has many components. Or if they cannot do that, they resort to using tests that can test many means at once--manova is an example.