5

Imagine a simple experiment, trying to answer a simple question. For example, is body temperature the same in men and in women ?

To answer this question, let's say you sample 10 men, and 10 women, randomly from a given city, and measure their respective body temperature (same protocol of measurement for everybody, of course).

Then imagine you get a significant (alpha=5%) difference between these two samples.

You cannot ignore a possible statistical fluke, can you ? (This may constitute a subsidiary question, and I will be pleased if you can answer it too, but the main question lies below) You may want to repeat this experiment a few times, for example, in independent cities, to get very confident about the reality of the difference you observed in the first experiment

Imagine again, that you repeat this experiment 8 times (including the first one), and you observe a significant difference between men and women in 4 of them.

My question is : How much confident can I be that the difference is real, if I have only this information : 4 out of 8 independent tests were significant at alpha = 5% ? (Or, to paraphrase, How can I calculate the overall p-value, when all I have is the p-value linked to each repetition experiment ? Maybe I need additional information ?)

(This is a simple example, for thinking efficiently about a real problem much more complicated...)

amoeba
  • 93,463
  • 28
  • 275
  • 317
Rodolphe
  • 861
  • 5
  • 16

1 Answers1

10

The p-value from each experiment should have a uniform distribution between 0 and 1 under the null hypothesis, so tests of the null hypothesis over all experiments can be based on this. Perhaps the most common test statistic is Fisher's: for p-values $p_j$ from $m$ independent experiments the negative log of each follows an exponential distribution

$$-\log p_j\sim \mathrm{Exp}(1)$$

and twice their sum a chi-squared distribution with $2m$ degrees of freedom.

$$-2\sum_j^m \log p_j \sim \chi^2_{2m}$$

So an overall p-value $p^*$ can be got from the chi-squared distribution function $F_{\chi^2}(\cdot)$:

$$p^* = 1-F_{\chi^2}\left(-2\sum_j^m \log p_j; 2m\right)$$

If you only know whether or not $p_j<\alpha$ the no. "successes" follows a binomial distribution with probability parameter $\alpha$ and sample size $m$:

$$\sum_j^m I(p_j) \sim \mathrm{Bin}(\alpha,m)$$ where the indicator function $$I(p_j)=\left\{ \begin{array}{ll} 0 & \text{when } p_j\geq\alpha \\ 1 & \text{when } p_j<\alpha \end{array} \right. $$ & so you can use the binomial distribution function $F_\mathrm{Bin}(\cdot)$ to calculate an overall p-value $$ p^*=1-F_\mathrm{Bin}\left(\sum_j^m I(p_j)-1;\alpha,m\right) $$

Read up on meta-analysis for more complicated situations, & for the (often more useful) estimation of an effect size measured over several studies, & for assessment of heterogeneity (are different studies really measuring the same thing?).

Scortchi - Reinstate Monica
  • 27,560
  • 8
  • 81
  • 248