2

I have a set of independent null hypothesis significance tests and their resulting p-values and I would like to make the claim that they are not all null by looking at the distribution of independent p-values. Since under the (meta) null hypothesis that all my hypotheses are actually null, the distribution of p-values is uniform, I though that I could simply perform a Kolmogorov-Smirnov test on that distribution and report the resulting p-value. Is this legitimate and is this something that any of you has seen done before? If so, would you have a reference?

I know of false discovery rate by Storey and Tibshirani but I would really wish to have a single p-value for the whole distribution rather than a set of q-values.

user1764944
  • 91
  • 2
  • 3
    I would advise approaching this cautiously. First, many tests do not produce exact p-values: they are approximate. Second, others do produce excellent p-values only when the p-values are small. Third, a test statistic with a discrete distribution obviously cannot yield a uniform distribution of p-values. Fourth, you're concerned about a specific alternative hypothesis, where the density of the p-value distribution grows at the lower values and shrinks at the higher, so this calls for a specialized test. The first three imply that a test of uniformity might give misleading results. – whuber Jun 29 '18 at 15:23
  • 1
    Have you seen https://stats.stackexchange.com/questions/20616? – whuber Jun 29 '18 at 15:23

0 Answers0