6

I have a bunch of independent $p$-values and now I want to combine them using the Fisher's method. However, in R, when a $p$-value is zero, the log(p-value) becomes $-\infty$, so that the test statistic $X^2$ is $\infty$. This will give the Fisher's method $p$-value of 0.

If I check the $p$-values' histogram, they look pretty uniform, so I expect that the Fisher's method should give me a not-so-small $p$-value (should not reject the null). Is there a way to get around this issue? How about the Stouffer's method? Will that method automatically take care of the issue when $p = 0$ (or close to 0)?

Update: I tried the Stouffer's method, but still the issue of $\infty$ is not solved...

amoeba
  • 93,463
  • 28
  • 275
  • 317
alittleboy
  • 743
  • 1
  • 9
  • 29
  • Can you provide information on how you are calculating p-values? – Sameer May 09 '13 at 03:14
  • @Sameer: I calculate p-values by Monte Carlo approach, i.e. p=(r+1)/(N+1). However, since I used two-sided p-value, it is possible to have p-value exactly equals 0, when r=N. – alittleboy May 09 '13 at 03:22
  • 3
    Isn't the p-value just less than some number, which is based on the number of simulations you did? E.g. if you did 1000 sims, the p-value given is 0, but it's just <0.001. So you treat it as 0.001. (Or whatever). – Jeremy Miles May 09 '13 at 03:34
  • 6
    @JeremyMiles When calculating bootstrap or Monte Carlo p-values if no test statistics as extreme as the observed statistic is found amongst the $B$ replicates, one generally sets the p-value to $\frac{1}{B}$ because the test statistic has been observed once (your observed test statistic!). However, this still doesn't account for the fact that you actually have a censored distribution i.e. you cannot observe p-values less than $\frac{1}{B}$. A quick Google search did not lead me anywhere; I wonder if someone else has thought about this problem. – Sameer May 09 '13 at 03:58
  • re: @Sameer Agree! You must increase what Sameer is calling $B$. If you instead use $\frac{1}{B}$ in cases of $p=0$, it should not increase type I error, but may increase type II error probability. – ndoogan May 09 '13 at 04:07
  • @JeremyMiles: for 2-sided test, a p-value can be exactly 0. My general question is that, whenever p-value=0 is present, how does the Fisher's test handle it? It will always reject the null since the test statistic is infinity – alittleboy May 09 '13 at 04:11
  • 1
    @alittleboy re: "when r=N". According to your equation, when $r$ and $N$ are equal, $p=1$. Doing a 2-sided test doesn't mean calculating $p$, then calculating $1-p$, here, does it? – ndoogan May 09 '13 at 04:15
  • 1
    @alittleboy, what are "r" & "N" here? You should not have any true 0's, even if you are doing a 2-sided test. – gung - Reinstate Monica Aug 12 '15 at 23:21

1 Answers1

1

Irrespective of the discussion in the comments about how these $p$-values of $0$ arose there are methods for combining $p$-values which can be calculated if $p=0$.

As the OP indicated neither Fisher's method nor Stouffer's works.

The method of Edgington based on the sum of $p$, the closely related mean $p$ method, the method using logit of $p$, Tippett's method based on the minimum $p$ and variants of Wilkinson's method of which Tippett is a special case can all be calculated. Whether that is a sensible thing to do depends on the scientific question of course.

All the methods mentioned are available in the R package metap which, disclaimer, I wrote and maintain.

mdewey
  • 16,541
  • 22
  • 30
  • 57