3

I understand the false discovery rate (FDR) is weakly less than the familywise error rate (FWE), and FDR is thus a less stringent way to control for type 1 errors. However, will a procedure that ensures $FDR\leq \alpha$ necessarily reject any hypothesis rejected by a procedure that ensures $FWE\leq \alpha$? The Benjamini and Hochberg (1995) method rejects at least as much as Holm (1979) and Bonferroni methods, but it is unclear to me if this would be true for arbitrary methods controlling FDR and FWE.

Golden_Ratio
  • 343
  • 7

1 Answers1

0

Ok I found that the Benjamini–Yekutieli (2001) procedure, which controls FDR under arbitrary dependence of p values, uses a lower cutoff than Benjamini and Hochberg (1995) and apparently lower than Holm, implying that some nulls rejected by Holm may get accepted by Benjamini–Yekutieli (2001).

Golden_Ratio
  • 343
  • 7
  • 1
    You probably want to read [this question and the most up-voted answers](https://stats.stackexchange.com/q/111756/44269), **and** the comments, before you get too excited about Benjamini-Yekutieli. The OP feels that no one there has come up with an intuitive example where Benjamini-Hochberg **would not** apply, but Benjamini-Yekutieli **would**. BY seems to me to be reaching for some kind construct which may not actually exist. Note: no accepted answer. – Alexis Dec 13 '21 at 02:31
  • @Alexis Thank you for the comment. But putting aside whether there is plausible setting where BY would apply and BH would not apply, on purely theoretic grounds, would you agree that BY may possibly reject fewer nulls than Holm? – Golden_Ratio Feb 10 '22 at 21:43
  • Oh, for sure. BY is quite conservative by comparison. I just do not know when it is appropriate to use it. (Pedantic aside: I never say "accept the null" [for reasons](https://stats.stackexchange.com/tags/tost/info). :) – Alexis Feb 10 '22 at 21:52
  • @Alexis Fair enough. The reason I was interested in this is because in the 1995 Benjamini Hochberg paper, they mention that since any procedure that controls FWE also controls FDR, "a gain in power may be expected" from an FDR controlling procedure. But this remark didn't seem to follow logically since strictly speaking, an FDR controlling procedure may still not reject as much as an FWE controlling procedure. – Golden_Ratio Feb 10 '22 at 22:33
  • I do not agree: what FWER applies under the conditions where BY applies? I think there's an apples to oranges there. – Alexis Feb 10 '22 at 23:20
  • @Alexis So do you think that the remark in the '95 paper that compares power among general FDR and FWE controlling procedures is not meaningful then? But I am not sure why you think it's apples and oranges; why can't we use Holm and BY in the same situations? They both control their respective error types under arbitrary dependence of p values. – Golden_Ratio Feb 11 '22 at 05:20
  • Under what circumstances would you use BY? Again: see the link in my first comment. I cannot think of a single example where **BH** does not control with greater power than Holm, and cannot think of a circumstance where Holm would apply, but BH would not. – Alexis Feb 11 '22 at 16:22