Understanding P-Value with respect to the Level of Significance

Question

The definition of P-value is the probability of obtaining a sample that is more extreme than the ones observed in your data, assuming that the null hypothesis is true. That 's very well-explained in this answer.

In a nutshell: $P(Test~Statistics~of~H_1 \geq Some Value_1 ~~|~~ True~Value~of~H0 = SomeValue_2)$

---

How I understand that, suppose $P_{value} = 0.06$, that means if we repeat the experiment on different samples 100 times, we expect to see the sample test statistics $\geq Some Value_1~$ 6 times out of 100. That means, the higher $P_{value}$ is, the more likely our alternative hypothesis is correct (and more likely we should reject null-hypothesis).

---

But what is confusing me is the rule that says, if $P_{value} \leq \alpha \implies Reject~H_0$. For example, if the level of significance is 0.06, that means $P_{value}$ should be less than 0.06 to accept the alternative hypothesis, which contradicts the first statement.

---

Any help in understanding it intuitively is very much appreciated.

Answer 1

It would be more of a comment than an answer but I need to cite you so I need an answer. I anticipate that I will provide an intuitive logical answer rather than a fully mathematical one, because I think it will be more effective based on the nature of the question.

You say:

How I understand that, suppose $P_{value} = 0.06$, that means if we repeat the experiment on different samples 100 times, we expect to see the sample test statistics $\geq Some Value_1~$ 6 times out of 100.

Ok. But you see that 6 times out of 100 GIVEN THAT THE TRUE VALUE IN THE POPULATION IS THE ONE INDICATED BY THE NULL HYPOTHESIS. So, deciding on the observed t statistic, you will be wrong 6 times out of 100 in repeated samples. So the higher the pvalue, the higher the probability that you draw the wrong conclusions when looking at the sample t stat in repeated samples. Or analogously, you can see the pvalue as INVERSELY proportional to the trustworthiness of high values of the t statistic in a generic sample. So the following

That means, the higher $P_{value}$ is, the more likely our alternative hypothesis is correct (and more likely we should reject null-hypothesis).

Is wrong, because by definition the p-value is the probability that you will make a mistake when looking at high absolute values of the t statistics. Indeed you will reject the null when the p-value is low, which means that, in repeated samples, you have a low probability of seeing a deceptive and misleading high value of the t statistic. Or, conversely, you have a high probability that, when the observed t-statistic is very high in absolute value, then this tell you the truth about the population.

Was it intuitive enough?

Answer 2

You have your interpretation of p-values backwards - the lower (i.e. closer to 0) a p-value is , the more significant it is, meaning we have more evidence to reject the null. A p-value of 0.06 means that with 100 replications of your experiment, you'd expect 6 of them to have a distribution as or more extreme than what you saw. The smaller this number is, the less likely it is that you'd see something so extreme under the null hypothesis. If that number is small enough (below your pre-set level of significance, typically 0.05), then the null hypothesis is so unlikely that you reject it.

If your p-value is 0.04, you have a 4% of seeing such an extreme sample by chance alone, which is good evidence to reject the null. If your p-value is 0.001, you have only a 0.1% chance of seeing such an exteme sample, which makes the null hypothesis even less likely. Smaller p-values are more significant, so if p is less than alpha, you reject the null.