1

Problem

If we test a true point Null hypothesis using a continuously distributed test statistic, it is possible to prove that the $p$-value resulting from such test is a realisation of a uniform distribution (see for example here and here). For these specific cases, the probability density function of the $\mathcal{U}(0,1)$ distribution offers a closed form expression of the distribution of $p$-values under the Null.

However, uniformity breaks down if we test true composite Null hypotheses (as pointed out by @whuber here and addressed in these questions here and here) or if we test false Null hypotheses. In these cases, as well as for cases in which a discrete test statistic is used, we can approximate the distribution of the $p$-values using repeated simulations of the same statistical test.

Now, I was wondering whether it is also possible to find a closed form or analytical expression of the distribution of $p$-values in any of these cases - and if yes, how. A solution to the following example would be enough to get me started, I think.

Concrete example

Let $X \sim \mathcal{N}(\mu,\sigma^2)$, $\sigma^2$ known. Let us then state the following two hypotheses:

$$\begin{align} H_0: \mu \leq \mu_0 \text{ vs } H_1: \mu > \mu_0 \end{align}$$

Let us furthermore assume that $\mu$ is strictly smaller than $\mu_0$, say $\mu = 0 < \mu_0 = 1$. Under these assumptions and using any test statistic that is feasible, what is the closed form or analytical expression of the distribution of the $p$-values?

Community
  • 1
DrosoNeuro
  • 135
  • 10
  • 1
    The answer is yes, because this is precisely the information used to compute the *power* of a test. That indicates just how broad this question is. Is there a specific problem you need to address? – whuber Mar 04 '19 at 17:58
  • @whuber I restated the question slightly so as to also include the question *what* the closed form expression is. I don't need it for a specific problem, but to gain a better general understanding of the distributional properties of p-values. A solution to example stated would already be enough to get me started, I think. – DrosoNeuro Mar 04 '19 at 22:09
  • To be honest, I don't think this is possible to compute without any further assumptions on $\mu$. The distribution of $p$-values will be different for any given true $\mu$, and it's impossible to know it without knowing the value of $\mu$. @whuber What did you have in mind when you answered "yes"? – amoeba Mar 04 '19 at 22:44
  • 1
    @amoeba I had in mind performing the computation for all $\mu.$ The result is a parameterized family of distributions. – whuber Mar 04 '19 at 22:47
  • @amoeba How would the computation work if we fix $\mu$ to a specific value. Say, $\mu_0 = 1$ and $\mu = 0$? – DrosoNeuro Mar 04 '19 at 22:59
  • I don't know (would have to think about it) but I am sure it's computable. If this is what you are asking then please edit to clarify. – amoeba Mar 04 '19 at 23:06
  • Done - any help would be greatly appreciated! – DrosoNeuro Mar 04 '19 at 23:13
  • Let mu_0=0, mu=-1, sigma=1, and sample size n. Then you can use z-test with z-statistic distributed as N(-1, 1/n). P-value is equal to 1-F(z), where F() is standard normal CDF. Not sure how much of a closed form you can get here... – amoeba Mar 04 '19 at 23:23

0 Answers0