0

I have 11 tests and I expect at least half of them to be truly different (I expect that from prior knowledge, there are papers that directly indicate possible difference). But the power of each test is not that big, resulting p-values are not that small:

0.0403, 0.0726, 0.0289, 0.6864, 0.003, 0.3539, 0.3753, 0.1256, 0.0292, 0.0858, 0.0024

Do I really need a FDR correction in this situation? Can I say that it could be too strict, select all tests that gave a raw p-value less than 0.05 (I am working in the field where people still appreciate this magic 0.05 number) and describe the prior knowledge on why we expect that many positive results?

I am not aiming for a lot of positive results, I still have 2 "significant" after FDR correction, however, I feel that smth is wrong with applying FDR when I expect a lot of true positive results (but have small power unfortunately).

German Demidov
  • 1,501
  • 10
  • 22

1 Answers1

1

The false discovery rate (FDR), unlike familywise error rate methods, the FDR does not assume that all null hypotheses are true, but that:

  1. some null hypotheses may be false (i.e. true positives), and
  2. for each null hypothesis that is rejected sequentially, the estimated probability of the next null hypothesis in the sequence being a true positive is updated (increased).

Because you are testing multiple null hypotheses $\alpha$ cannot mean the probability of falsely rejecting a single null hypothesis as it can for a single hypothesis test. Therefore, you should adjust for multiple comparisons.

I would go farther, and say that you should conduct relevance tests (see, for example my answer), to explicitly parse out whether any negative results are because the differences are equivalent or trivially different, or simply because your test is under-powered. (You would still apply the FDR when conducting relevance tests.)

Alexis
  • 26,219
  • 5
  • 78
  • 131