2

Doing multiple comparisons increases the number of false positive findings. In the Bonferroni method, the adjusted critical level is $\hat{\alpha} = \frac{\large\alpha}{N}$ where $N$ is the number of tests. However, demanding this from each test increases the number of false negatives.

In the Holm-Bonferroni and FDR methods, the smallest p-values have to be likewise below $\frac{\large\alpha}{N}$. Is this some general property?

In order to control the number of false positives, when doing $N$ tests, do the smallest $p$ have to be smaller than $\alpha/N$? It seems the Holm-Bonferroni and FDR differ only with respect to the following values.

user132576
  • 21
  • 1

1 Answers1

0

Simply put, no, it is not a general property of familywise error rate (FWER) control or of false discovery rate (FDR) control that the smallest p-value must be smaller than α/N.

The Holm-Bonferroni procedure is a "step-down" method, so the significance of each p-value depends on the lowest p-value. However, that is not the case when using "step-up" methods, such as the Hochberg procedure (for controlling the FWER) and the Benjamini-Hochberg procedure (for controlling the FDR).

For example, say you conduct 3 tests and get the following p-values: .02, .03, and .04. None of those p-values are < α/N (assuming α=.05), yet all of them would be significant using a step-up procedure such as the ones I mentioned.

Bonferroni
  • 329
  • 2
  • 7
  • Other exceptions (just to mention a FWER controlling one) would be Dunnett's test (and related test such as the one by Dunnett and Tamahane), where the threshold is a little bit higher than $\alpha/N$, weighted Bonferroni(-Holm) tests, various closed testing procedures etc. Just in case the list above may have given the impression that that the smallest p-value must be smaller than $\alpha/N$ for FWER controlling methods. – Björn Mar 01 '17 at 13:13
  • As I noted, the Hochberg method (not to be confused with the Benjamini-Hochberg method) is for controlling the FWER. But you are right that there are numerous other FWER-controlling methods that don't require p < α/N for the lowest p-value. Dunnett's is a good example. – Bonferroni Mar 01 '17 at 21:19