8

I am using Fisher's combined test to fuse several different independent tests. I have a problem understanding the results in some cases.

Example: Let's say I run two different tests, both with the hypothesis that mu is smaller than 0. Let's say that n is identical and the two samples have the same calculated variance. However, let's assume that one test yielded an average that is $1.5$ and the other is $-1.5$. I will get two complementing p-vals (e.g., $0.995$ & $0.005$). Interestingly, combining the two brings about a significant $p$-value in the Fisher test: $p=0.0175$.

This is weird because I could have chosen the exact opposite test $(\mu>0)$ and sampled results - and still get $p=0.0175$. It's almost as if the Fisher test does not take the direction of the hypothesis into account.

Can anyone explain this?

Thanks

amoeba
  • 93,463
  • 28
  • 275
  • 317
  • 2
    If I interpret this question correctly, the discussion in Rice, [*A Consensus Combined P-Value Test and the Family-wide Significance of Component Tests*](http://www.lifesci.ucsb.edu/eemb/faculty/rice/publications/pdf/23.pdf) (Biometrics 1990) explains this problem: see p. 304. The paper offers a solution. – whuber Jun 25 '12 at 13:08
  • Actually using Fisher's combined probability test the combined p for 0.995 and 0.005 is 0.03. Not that it changes the interpretation (smile) but I am wondering where the 0.0175 came from. – AussieAndy Jan 19 '16 at 23:35
  • @AussieAndy Yes, I agree -- I make it about 0.03136 – Glen_b Jan 20 '16 at 01:10

2 Answers2

10

The Fisher combination test is intended to combine information from separate tests done on independent data sets in order to obtain power when the individual tests may not have sufficient power. The idea is that if the $k$ null hypotheses are all correct the $p$-value will be uniformly distributed on $[0,1]$ independently of each other. This means that $-2 ∑ \log(p_i)$ will be $\chi^2$ with $2k$ degrees of freedom. Rejecting this combined null hypothesis leads to the conclusion that at least one of the null hypotheses is false. That is what you are doing when you apply this procedure.

Michael R. Chernick
  • 39,640
  • 28
  • 74
  • 143
  • 1
    This does not seem to address the real issue raised by the question: because the two p-values are *symmetrically opposite,* and therefore (at least according to some intuition) ought to "cancel," how is it that Fisher's method produces a "significant" result--and which conclusion does it support?? – whuber Jan 19 '16 at 23:42
  • 2
    That should be $2k$ df. – Glen_b Jan 20 '16 at 01:15
  • 2
    +1 for *Rejecting this combined null hypothesis leads to the conclusion that* at least one *of the null hypotheses is false.* – eric_kernfeld Dec 08 '19 at 19:22
  • 1
    I think the OP & at the time @whuber in his comment are misunderstanding the meaning of the rejection of the combined null hypotheses. eric_kernfield is emphasizing this by repeating what I said in my answer. – Michael R. Chernick Dec 08 '19 at 19:28
  • @Michael, I doubt I misapprehended something as elementary as what it means to reject the combined hypotheses. What is missing from your answer is an *explanation* of the apparent paradox raised by the OP and in my comment. One place we might seek an explanation is to note that in one case the data were consistent with the null and in the other case they were noticeably inconsistent. The combined dataset thereby still exhibits some inconsistency with the null, which may be why the Fisher p-value is low--but not that low. This deserves thought and study rather than casting aspersions. – whuber Dec 08 '19 at 21:53
  • 1
    I agree with that. I wasn't casting aspersions. I just disagreed with your original remark that seemed to suggest that I was missing something in my answer. I appreciated Eric Kernfeld's remark. I still think the OP misinterpreted the meaning of the null & alternative hypotheses for Fisher's combination test. Maybe you didn't misunderstand but your comment seemed to suggest something that you might not have intended. – Michael R. Chernick Dec 08 '19 at 22:10
3

There are several ways of combining $p$-values and some of them have this property and some do not. This is partly because the problem is not well specified. There has been an extensive simulation study of many of the most-well known methods. The bottom line is that if you want the property of cancellation you can have it but you do not have to.

mdewey
  • 16,541
  • 22
  • 30
  • 57