Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [combining-p-values]

Methods for combining p-values from several hypothesis tests with the same null hypothesis into one aggregate p-value, such as e.g. Fisher's method and Stouffer's method.

Techniques for combining the outcomes of several independent hypothesis tests with the same null hypothesis into one aggregate outcome. There exist various methods that take a set of $p$-values from individual tests and yield one aggregate $p$-value; two most common ones are Fisher's method and Stouffer's method.

See also .

76 questions
57
votes
3 answers

When combining p-values, why not just averaging?

I recently learned about Fisher's method to combine p-values. This is based on the fact that p-value under the null follows a uniform distribution, and that $$-2\sum_{i=1}^n{\log X_i} \sim \chi^2(2n), \text{ given } X \sim \text{Unif}(0,1)$$ which I…
Alby
  • 2,103
  • 3
  • 19
  • 22
30
votes
4 answers

Does a uniform distribution of many p-values give statistical evidence that H0 is true?

A single statistical test can give evidence that the null hypothesis (H0) is false and therefore the alternative hypothesis (H1) is true. But it cannot be used to show that H0 is true because failure to reject H0 does not mean that H0 is true. But…
Leander Moesinger
  • 421
  • 5
  • 14
30
votes
3 answers

Can a meta-analysis of studies which are all "not statistically signficant" lead to a "significant" conclusion?

A meta-analysis includes a bunch of studies, all of which reported a P value greater than 0.05. Is it possible for the overall meta-analysis to report a P value less than 0.05? Under what circumstances? (I am pretty sure the answer is yes, but I'd…
Harvey Motulsky
  • 14,903
  • 11
  • 51
  • 98
24
votes
4 answers

Stouffer's Z-score method: what if we sum $z^2$ instead of $z$?

I am performing $N$ independent statistical tests with the same null hypothesis, and would like to combine the results into one $p$-value. It seems that there are two "accepted" methods: Fisher's method and Stouffer's method. My question is about…
amoeba
  • 93,463
  • 28
  • 275
  • 317
13
votes
3 answers

R package for combining p-values using Fisher's or Stouffer's method

Is there an R package (or even a base R function) that implements Fisher's or Stouffer's method for combining p-values? Coding this should be almost trivial, but I'd rather use (and cite) a package. Example code in this question: Fisher's method for…
krlmlr
  • 749
  • 1
  • 8
  • 35
10
votes
4 answers

Why does Stouffer's method work?

It seems like a fairly straightforward question, but when I really think about it, Stouffer's method doesn't make sense to me. This is why: Assume a two-tailed hypothesis. You first calculate $z_i$ from $p$-values. So let's take a fairly simple…
will
  • 101
  • 1
  • 3
9
votes
1 answer

Why is my combined p-value, obtained using the Fisher's method, so low?

I have multiple individuals, for whom I collected two time series of some parameter. For each individual, I calculated, whether these time series are correlated. So, if I have 20 individuals, as a result I have 20 rho and 20 p-values. Then, I would…
Dmitry Smirnov
  • 635
  • 1
  • 6
  • 11
8
votes
2 answers

Understanding Fisher's combined test

I am using Fisher's combined test to fuse several different independent tests. I have a problem understanding the results in some cases. Example: Let's say I run two different tests, both with the hypothesis that mu is smaller than 0. Let's say that…
Eitan Schechtman
  • 81
  • 1
  • 2
7
votes
2 answers

Test for significant excess of significant p-values across multiple comparisons

I have what feels like a simple question, but was unable to find answers easily. The situation Let's say I have a gene microarray dataset with tens of thousands of genes and small (<100) number of samples. I am interested in simple mean differences…
Karolis Koncevičius
  • 4,282
  • 7
  • 30
  • 47
7
votes
1 answer

What are the reasonable properties of p-value combination methods when all p-values are the same?

Suppose we are testing a null hypothesis $H_0$, by combining evidence from $n$ independent tests, using a p-value combination method $M$. The combined p-value is $M(p_1,...,p_n)$. Assume the extreme case when we get the same p-value from all…
yliueagle
  • 755
  • 2
  • 6
  • 10
7
votes
1 answer

What does inverse-chi-square in Fisher method (classifying) exactly do?

In the book “Programming Collective Intelligence” Segaran explains the Fisher method for categorizing text as an alternative to Naive Bayes classifier. The Fisher method uses inverse-chi-square-distribution, which I do not really understand. I…
aufziehvogel
  • 203
  • 1
  • 5
6
votes
2 answers

Why does Fisher's method yield $p\gg 0.5$ when combining several p-values all equal to $0.5$?

W = 0 for p in xrange(10): W += -2*log(0.5) print "W:",W final_p = 1 - scipy.stats.chi2.cdf(W, 2*10) print "Final p-value",final_p Why does this result in a final p-value of 0.84, when I used only the most insignificant p-values (p=0.5 for the…
317070
  • 313
  • 1
  • 8
6
votes
1 answer

Fisher's method for combing p-values - what about the lower tail?

I have a bunch of independent p-values and now I want to combine them using the Fisher's method. Each of the individual p-values is coming from a one-sided test. I am just a little bit confused about the "side" of the Fisher's method test, i.e. when…
alittleboy
  • 743
  • 1
  • 9
  • 29
6
votes
1 answer

Fisher's method of combining p-values when one of the p-values is zero

I have a bunch of independent $p$-values and now I want to combine them using the Fisher's method. However, in R, when a $p$-value is zero, the log(p-value) becomes $-\infty$, so that the test statistic $X^2$ is $\infty$. This will give the Fisher's…
alittleboy
  • 743
  • 1
  • 9
  • 29
6
votes
0 answers

Can one use Stouffer's method to integrate t-values?

I would like to integrate T-statistics values from multiple, independent tests. How can this be done? Since my degrees of freedom are many ($n>1000$) I was thinking to use the Stouffer's method, which is really for Z-statistics score. However,…
Federico Giorgi
  • 189
  • 1
  • 11
1
2 3 4 5 6