0

I'm almost positive I'm going to be marked for duplicate, but I really really just need this sorted out once and for all with a simple example.

I don't understand how a small p-value is evidence against the null hypothesis.

Take a simple example: testing if a coin is fair. This is modeled with a binomial distribution with $n$ fixed (the number of times we flip the coin) and parameter $p$. The null hypothesis $H_0$ is $p = 0.5$.

Now, let our test statistic be $D = |X - np|$, where $X$ is the number of heads in the sample (binomially distributed). The p-value is defined as $p = P(D \ge d\ |\ H_0)$, where $d$ is our observed value of the statistic.

Lets say we find, initially, for $100$ tosses that $60$ heads occurred. So $d = 10$. Then $p = P(D\ge 10\ |\ H_0)$.

Here begins my interpretation:

Then a small p-value means its unlikely for D to be greater than 10, assuming our hypothesis is true.

This means, its more likely for $D \le 10$, which is "near" $0$. That means that the sample $X$ is very likely close to a binomial distribution with p=.5.

That's my innterpretation of it, for which I can't find anything wrong. Why, then, does a small p-value indicate strong evidence against the hypothesis?

That makes zero sense to me. If $p$ is small, then our test statistic is close to zero, meaning our model is mostly likely correct! Why on Earth is that evidence against $H_0$, that's completely in support of it! Please, what is wrong with this? I can't for the life of me tell.

user3002473
  • 145
  • 8
  • See https://stats.stackexchange.com/questions/tagged/p-value?sort=votes for more about this subject. – whuber Apr 18 '18 at 15:44
  • @whuber I have looked at the question you marked mine as a "duplicate" of, but I have already read it and it does _not_ answer my question. In fact, the accepted answer only serves to _enforce_ my faulty interpretation. – user3002473 Apr 18 '18 at 15:50
  • 1
    "If p is small, then our test statistic is close to zero, meaning our model is mostly likely correct!" This is the source of your mistake. If n is close to 50 then the p-value will be close to 50%. If $D\ge{10}$ then it is far away and the p-value is small. The p-value on a two tailed test will be 5.6% at $D=\pm{10}$. – Dave Harris Apr 18 '18 at 16:02
  • So in this _specific_ example, the p-value isn't actually small, because our sample isn't actually statistically "close" to the hypothesized model? – user3002473 Apr 18 '18 at 16:06
  • There are many good answers to the duplicate that address your questions. Whether or not you consider my answer there to be any good, I discuss these issues *thoroughly* and *explicitly* in that answer. – whuber Apr 18 '18 at 16:08
  • 1
    @whuber I didn't realize that one was yours. To be fair I stopped near the top because it's a long read, and I have a short millennial attention span. – user3002473 Apr 18 '18 at 16:10
  • 1
    That's _millennial_ to non-millennials.... – Nick Cox Apr 18 '18 at 16:12
  • 2
    Case in point, couldn't even be bothered to look up the correct spelling. – user3002473 Apr 18 '18 at 16:15
  • 2
    (+1) for cando[u]r. – Nick Cox Apr 18 '18 at 16:23

0 Answers0