1

According to this thread

What is the meaning of p values and t values in statistical tests?

a p-value essentially lets me compare the probability of tail-end (unlikely) events occurring and judging that against the significance level $\alpha$ to decide whether the null hypothesis should be rejected. In the example on this thread, Muriel Bristol visits Fisher and it essentially asks what's the probability of getting 5/6 correct guesses given the probability of her guessing correctly is $0.5$.

In the context of this, p-values make sense if I assume an unlikely event already happened. Because if such an event happens, then interpreting the significance of that event for a null hypothesis is a natural way to decide whether that hypothesis has been rejected.

My Question

Do p-values assume an unlikely event has already happened? If so, why is this part of the definition? If not, how does knowing the p-value tell me anything about why I should reject the null hypothesis?

Without knowing an unlikely event occurred, it would seem all a p-value can say is "if an unlikely event occurs, then the null hypothesis would be rejected."

Community
  • 1
Stan Shunpike
  • 3,623
  • 2
  • 27
  • 36
  • 3
    I don't get your assertion "p-values make sense if I assume an unlikely event already happened"; can you explain your reasoning? – Glen_b May 05 '16 at 04:37
  • Suppose I am considering whether a die is biased. If I roll this die several times and get the same number each time, that would give me a lot of information about the bias of the die, right? So the occurrence of an unlikely event (or cluster of them) tells me something about the system. So if my null hypothesis is the die is fair, and I roll all 6s, then I would reject that null hypothesis because the pvalue of getting all 6s would be lower than the significance level $\alpha = 0.05$. Did that make any sense? – Stan Shunpike May 05 '16 at 04:48
  • That doesn't explain why the p-value would not make sense if the observed event was less unusual. – Glen_b May 05 '16 at 04:49
  • I think I do not understand what frequentist statistics means and therefore I do not understand this way of thinking. – Stan Shunpike May 05 '16 at 05:02
  • @StanShunpike the classic hypothesis testing you are describing (which is the one working with p-values) is also called "frequentist statistics". It is opposed to Bayesian statistics, which looks at the probability that event B will happen given that another event A has already happened. If you have never done Bayesian (which is rarely taught in introductory courses) you can skip that caveat and just look at the rest of the answer. – rumtscho May 05 '16 at 09:19

3 Answers3

5

In frequentist statistics, it does not matter whether an event has occurred or not. P-values give an idea of how often the data would be observed given the null hypothesis is true. If you get a p-value of .15 then we would say that, if the null hypothesis is true, the data or more extreme data would occur in 15% of an infinite amount of cases.

P-values are of limited utility in general, see the statement be the American Statistical Association for more information.

Stan Shunpike
  • 3,623
  • 2
  • 27
  • 36
Chris P
  • 132
  • 5
2

Do P-values assume that an unlikely event has already happened? No. The P-value is calculated by setting a null hypothesis within the chosen statistical model, but by the P-value itself makes no assumptions. However, if an analyst chooses to accept the null hypothesis despite finding a low P-value then, assuming the P-value is appropriately conditioned for the inferential task at hand, the analyst would be assuming that an unlikely event had occurred.

This answer is not intended to be just nit-picking about the agency of P-values versus analysts. It is necessary to keep the various components of statistical inference clearly labelled to avoid the standard confusions.

P-values have been "defined" in several different ways, and they take subtly different meanings within different statistical frameworks. The unlikely event thing comes from an interpretation of P-values, or a strategy for interpreting them as originally mentioned by Fisher.

Knowing the P-value tells you how unusual the data look according to the statistical model with the particular null hypothesis used. If the data look unusual then you have a reason to suspect that either the model is wrong, the null is wrong, both, or that an unusual event has occurred.

I agree with Chris P that you should have a look at the ASA statement on P-values, but you probably also should look at some of the references cited within. Among them, my own paper on the necessary distinction between hypothesis tests yielding dichotomous decisions and significance tests yielding a P-value to be interpreted as evidence is (of course!) well worth reading: http://www.ncbi.nlm.nih.gov/pubmed/22394284

Michael Lew
  • 10,995
  • 2
  • 29
  • 47
0

As stated in other answers, $p$-values don't assume that an unlikely event occurred. The $p$-value is a probability that is conditioned only on the null hypothesis being true, $$ P(\text{Obtaining data as or more extreme than that observed} \, | \, H_0) $$ and not on anything about the observed data.

Some people's logic concerning $p$-values, though, can be reduced to a triviality about the fact that data that were extreme under $H_0$ were observed (see rejoinder 6 in Berger & Delampady, 1987). The logic goes

Either an extreme event occurred and $H_0$ is true OR $H_1$ is false

And this is used as evidence against $H_0$. Though if we are careful we should in fact say

Either an event that would be extreme under $H_0$ occurred and $H_0$ is true OR an event that would be extreme under $H_0$ occurred and $H_1$ is false

which just reduces to

an event that would be extreme under $H_0$ occurred

In fact, this very fallacy is repeated in an answer to your linked question

innisfree
  • 1,124
  • 6
  • 23