0

I first summarize my understanding of p-values. We have an hypothesis $H_0$ (null hypothesis) that we want to test.

Now we build a test statistics $T$, a random variable. Depending on $H_0$ being true or not $T$ has a different distribution.

The p-value $P$ is then the statistics $P=F_T(T)$, if $F_T$ is the c.d.f. of $T$ given that $H_0$ is true.

It is well known that $F_X(X)$ is uniformly distributed for every r.v. $X$. In particular we have:

$P \sim U(0,1)$ if $H_0$ is true

This permits than to say that, if the p-value is small, e.g. $P \in [0,\epsilon]$, it will an unprobable value and than we can reject the null hypothesis.

So my questions are:

  • if this is really it, we could check if $P \in [a,a+\epsilon]$. Of course we cannot cherry pick $a$ after the $P$ value computation, but would it still be reasonable if all the world used $a=0.5$ instead of $a=0$ ?

  • Each statistics $T$ defines a reasoanable $P$ value. What are the criteria that are usually used for choosing the test statistics $T$ ?

  • Do the answers of the previous points involve maybe defining an alternative hypothesis $H_1$ and tuning $T$ so that if $H_1$ is true, $P$ is close to zero ? In this case, could $H_1 = not H_0$ be a reasonable alternative hypothesis ?

Thomas
  • 623
  • 3
  • 14
  • The p-value is not always $F(T)$. There can be upper-tailed tests, two-sided, and bioequivalence types of tests. The p-value is not always U(0,1) under the null hypothesis. Think about flipping a coin 10 times and testing null hypothesis prob. heads less than or equals 0.5. You can only observe 11 different possible number of heads and there are only 11 different p-values possible. – John L Feb 07 '21 at 17:52
  • https://stats.stackexchange.com/questions/10613/why-are-p-values-uniformly-distributed-under-the-null-hypothesis#:~:text=The%20p-value%20is%20uniformly,all%20other%20assumptions%20are%20met.&text=We%20want%20the%20probability%20of,comes%20from%20a%20uniform%20distribution. – Thomas Feb 07 '21 at 18:31
  • @John L you may be correct but does it mean that the whole answer above is just concerned with left tail test? And it does not apply in the majority of cases? Is this what you are suggesting ? – Thomas Feb 07 '21 at 18:33
  • Right. The p-value can be F(T), 1-F(T), 2 times min( F(T), 1-F(T)), etc. The p-value is not uniform when the distribution is discrete. Criteria for selecting test statistics and critical values are based on type 1 error rate and power. Neyman Pearson formulation shows in some cases there is a uniformly most powerful test and it tells us how to find the statistic and critical region in those scenarios. – John L Feb 07 '21 at 18:43
  • 1
    ["Here"](https://stats.stackexchange.com/search?q=%5Bp-value%5D+user%3A53690+is%3Aquestion+-%5Bcritical-value%5D) are a few questions regarding the definition and the intuition behind the p-value. Perhaps you will find them helpful. – Richard Hardy Feb 07 '21 at 18:53
  • Thanks if I find answers there to my doubts before an answer here I will delete the question – Thomas Feb 07 '21 at 18:55
  • @RichardHardy Thanks I am slowly reading through the questions and the answer you linked and they are really helpful. You posted many doubts that also I had in my mind and were ( and still are ) interfering with my understanding of p-values . – Thomas Feb 08 '21 at 14:18
  • Here for example https://stats.stackexchange.com/questions/31/what-is-the-meaning-of-p-values-and-t-values-in-statistical-tests/130772#130772 in the "dialogue between the teacher and the thoughtful student" it looks that the definition of p-values suggests THE CHOICE for the statistics $T$, which is something like $T(X)=p_{ALT}(X) / P_{H0}(X)$ ( focusing for now in the case of a simple alternative ) – Thomas Feb 08 '21 at 14:36
  • @Thomas, yes, this is the likelihood ratio test which is known to be the most powerful under some assumptions. My intermediate conclusion about $p$-values is that the definition as formulated in words is ambiguous and misleading. The formulas are typically OK, but their translation into words is not direct/intuitive. Unless both $H_0$ and $H_1$ are point hypotheses, there are problems with the interpretation. But I am not good at hypothesis testing, so my opinion might not be worth much. – Richard Hardy Feb 08 '21 at 17:32

0 Answers0