3

As we know, $p$-values are uniformly distributed under $H_0$. This urges me to ask if this constitutes a valid (re-)definition of the $p$-value.

$p$-value - A statistic with a uniform distribution on $[0, 1]$ under $H_0$.

Intuitively it makes sense for continuous distributions, but for discrete distributions, some adjustments are to be made. For example, this patched version might work,

$p$-value - A statistic $X$ such that $\forall c \in \left\{P(X = c|H_0) > 0\right\}$, $P(X \le c|H_0) = c$.

I'm asking this question because I have some trouble understanding the "power" of a hypothesis test. If this argument holds true, then I can continue to define the power as a function of its corresponding $p$-value statistic, which would be cool.

nalzok
  • 1,385
  • 12
  • 24
  • 1
    For discrete distributions (e.g., a $\chi^2$ test or a binomial test), only certain p-values are possible (see: [Comparing and contrasting, p-values, significance levels and type I error](https://stats.stackexchange.com/a/33500/7290)). – gung - Reinstate Monica Mar 31 '19 at 01:29
  • An uniform r.v. independent to your test will always satisfies your definition. – Francis Mar 31 '19 at 01:41
  • @Francis Actually this fact is part of my motivation! I mean, *technically* it can be considered a $p$-value; it just have extremely low power. I even think it would be a good idea to make it a reference point with “zero power”, and define the power of other $p$-values with respective to it. – nalzok Mar 31 '19 at 01:48
  • 1
    P-value under the null hypothesis is UNIF(0,1) only for test statistics that are continuous and exact. – BruceET Mar 31 '19 at 06:17
  • @BruceET Thanks for pointing that out, but I wonder if it’s fair to say the $p$-value of discrete/approximate statistics have an “approximately” UNIF(0, 1) distribution? Like, for discrete distributions, I might define a similar construct with a generalized inverse. – nalzok Mar 31 '19 at 06:28
  • Uniform dist'n of P-values in many circumstances is sometimes a useful and enlightening fact, but I think it is best to leave the definition of P-value as it is (because properly understood it makes intuitive sense). – BruceET Mar 31 '19 at 06:37
  • @BruceET I totally agree that the current definition is intuitive, but defining it as "the probability of obtaining a test statistic at least as extreme as the one we actually observed" sounds a bit too subject to me. I mean, how do you define "extreme" anyway? In some situations, it might be nice to have this important concept rigorously defined so that theoretical analyses could be made easier. This is just my attempt :) – nalzok Mar 31 '19 at 06:45
  • 'Extreme' means extreme in the direction(s) of the alternative hypothesis. // You are, of course, entitled to your opinion what is useful, but I would not want to tamper with the definition of P-value in this way. There is a distinction between a definition and a theorem (with conditions). – BruceET Mar 31 '19 at 07:51

0 Answers0