Multiple comparison problem

Question

If I conduct 19 tests (e.g. 19 pairwise comparisons of treatment means in an experiment), what is the probability that I will find at least one P value < 0.05 due to chance alone? With number of pairwise comparisons given by k*(k-1)/2 where k is the number of treatment levels, how does one calculate this?

Answer 1

I found an answer here, which says that the chance also depends on the level of significance you have set.

The chance of committing at least one Type I error for a set of comparisons can be estimated as c * .05, where c is the number of comparisons made. For our example with three conditions, we can make three comparisons, this would be 3 * .05 = .15, or a 15% chance of making at least one Type I error.

Since Type I error is getting p-value small even if it should not be.

In statistical hypothesis testing, a type I error is the rejection of a true null hypothesis (also known as a "false positive" finding or conclusion)

Answer 2

It depends on the correlation of your 19 tests and the distribution of your $p$-values. For independent tests and uniformly distributed $p$-values (see Why are p-values uniformly distributed under the null hypothesis? and comments) the chance for no $p$-value below 0.05 would be $$\prod_{i=1}^{19} P(p_i<0.05)=(0.05)^{19}$$ and therefore the answer to your question regarding the complementary event is $$1-(0.05)^{19}.$$

If the tests have a correlation of 1, the probability would only be 0.05.

If you don't know the correlation, the highest possible probability would be $19*0.05$.

With number of pairwise comparisons given by k*(k-1)/2 where k is the number of treatment levels, how does one calculate this?

• If you have k treatment levels, you can compare the first one to k-1 other levels.
• The next you compare to k-2 (because you already compared it to the first one).
• The next after that to k-3.
• And so on ... till you compare the last but one to the last.

So what's the sum from 1 to k-1? That's the well know formula from Gauß: $$\sum_{i=1}^{k-1}i=\frac{(k-1)k}{2}.$$