6

Hypothesis testing seeks to reject a null hypothesis ($H_0$) on the basis of an assumption made about the sample following a certain distribution. This assumption is conditional on $H_0$ being true. So what is the flaw with the following reasoning?

If I reject $H_0$ in favor of the alternative hypothesis ($H_1$), I must then assume that (1) $H_1$ is true, and (2) that at the same time the variable under examination follows the distribution used to reject $H_0$. Since we also stated that the assumed distribution is conditional on $H_0$ being true, it appears that $H_0$ has got to be true in order to justify the assumption made about the distribution. Logically, it would therefore seem that both $H_0$ and $H_1$ have to be true, which is a paradox, because they are mutually exclusive!

Silverfish
  • 20,678
  • 23
  • 92
  • 180
Constantin
  • 1,117
  • 1
  • 9
  • 24
  • 1
    This argument doesn't even make any sense except when $H_0$ and $H_1$ are *simple* hypotheses: that is, when each designates a *unique* distribution. That is a rare circumstance. Although in that case your argument at least makes sense, it is logically incorrect: one may assume *anything* for the sake of logical argument without asserting its truth. – whuber Nov 14 '14 at 22:52
  • 1
    Yeah our "traditional" hypothesis testing is strange. Try looking into the Bayesian version of hypothesis testing. You might find something more satisfactory. – Lauren Goodwin Nov 14 '14 at 21:52
  • @whuber I think this question actually has value as addressing a "common misconception". This is a type of argument that I've seen students come up with before. – Silverfish Nov 14 '14 at 23:35
  • 1
    @Silver I don't disagree--for otherwise I would have downvoted or closed the question and I have done neither. Some of the most interesting questions are those that exhibit different understandings of basic concepts, because they can reveal fundamental ideas. I am, however, generally concerned about the possibilities of people misreading or misunderstanding questions. The potential for this increases sharply when the question statement contains obvious errors or makes no sense at all. The purpose behind most of my comments on this site is to forestall any further misunderstanding. – whuber Nov 14 '14 at 23:43
  • 1
    @whuber You do a very valuable service! With a question like this I'd be very tempted to edit and reframe it as "what is the flaw with this reasoning?" That would still reflect the original query but by making it less assertive would be less likely to mislead later readers who might mistake this confusion for a serious controversy. (And goodness knows, we already have plenty of controversy and confusion about hypothesis testing...) – Silverfish Nov 15 '14 at 14:03
  • 2
    Bearing in mind the title suggests this question is part of one of the more active controversies about hypothesis testing, perhaps it should be retitled more specifically. "If we disbelieve $H_0$, why quote a p value calculated assuming $H_0$ was true?" seems to capture the crux of it? – Silverfish Nov 15 '14 at 14:07

3 Answers3

10

You have a misunderstanding. The 'alternative' hypothesis ($H_1$) is simply the negation of the null hypothesis. When conducting, say, a power analysis, we will specify a specific sampling distribution around a point estimate (for example a mean treatment effect) that we believe in, but rejecting the null does not make that point estimate true. Based on the logic of hypothesis testing, the alternative hypothesis is not that point estimate, it is just the negation of the null. There is no particular sampling distribution of a test statistic that is associated with the negation of the null.

In addition, the meaning of the $p$-value is predicated on what may well be a counterfactual premise. The $p$-value is the probability of getting a test statistic as far away from your null point value for your parameter (or further) if that point value were true, whether it is actually true of not. Even if the null isn't true, it can be true that the test statistic would have the specified distribution under the null.

You are striking on an important insight, though. Once you no longer believe the null obtains, it is no longer clear what meaning the $p$-value has to offer.

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
  • If the statistic has some distribution under the null (like $\chi^2$, F, or even Student-t distribution), in many cases, when the null is false, the statistic follows a "non-central" version of that distribution, with a non-centrality parameter that increases as the null grows more false. I learned about this from some posts by econometrician David Giles on his blog *Econometrics Beat*. – dimitriy Nov 14 '14 at 22:27
  • @DimitriyV.Masterov, it's true that it should follow *some* non-central distribution. There will be a specific non-central $t$ for $μ=\delta$, where $δ$ is some specific (non-0) value, but there isn't such a thing as a non-central $t$ for $μ≠0$. – gung - Reinstate Monica Nov 14 '14 at 22:49
  • @gung, so the assumption made about the distribution of the sample is purely theoretical so that even after rejecting the null hypothesis it is still assumed that my variable _would_ follow the assumed distribution if $H_0$ were true? – Constantin Nov 14 '14 at 22:54
  • @Constantin, yes: the assumption is that your test statistic would follow the assumed distribution if $H_0$ were true. This assumption (may) hold, whether the null is true or not. (Actually, there are lots of cases where the sampling distribution would diverge under the null, but that's a different story.) – gung - Reinstate Monica Nov 14 '14 at 22:59
  • 1
    "Once you no longer believe the null obtains, it is no longer clear what meaning the p -value has to offer." Is this a little strong? I do see your point. But *p* isn't denuded of all meaning just by being counterfactual. So interpretation as "evidence against $H_0$" still holds up ok even if we don't think it's the "correct" probability. Conditioning on something we believe false is a common (albeit often confusing) form of human reasoning, not a special perversity of statisticians! – Silverfish Nov 15 '14 at 00:28
  • @gung, if the assumption that my test statistic follows the assumed distribution may hold regardless of whether $H_0$ is true or not, on what basis can I reject $H_0$? – Constantin Nov 15 '14 at 01:08
  • @Silverfish, do you have any other examples of what you call _conditioning on something we believe is false_? – Constantin Nov 15 '14 at 01:10
  • 2
    @Constantin - yes, in everyday life we use such reasoning regularly. Suppose my neighbour drives to work 95% of the time and walks to work 5% of the time. If I look out and see his car on the drive, I may well think to myself "I think he hasn't gone to work today, because *if he had gone to work* (even though I don't believe it) he would probably have taken the car, but the car is still there". If you think about it, you can probably find other examples in daily life where you use similar reasoning. – Silverfish Nov 15 '14 at 01:20
  • 1
    @Constantin, we reject the null because our results are too improbable under the null. The fact that the null distribution would hold if the null were true does not undermine that. – gung - Reinstate Monica Nov 15 '14 at 02:25
9

(Started as a comment, but it's much too long)

Let's consider this a different way. A more general version of the question is --- can we use reasoning involving conditional probabilities when the thing we condition on is false?

It's not simply permissible -- it's necessary.

Consider this in the context of Bayes theorem:

$$P(A_i|B) = \frac{P(B|A_i)\,P(A_i)}{\sum\limits_j P(B|A_j)\,P(A_j)}$$

Note that the $A_j$ are mutually exclusive (and exhaustive). All but one of the conditionals in the denominator must pertain to a condition that doesn't hold - but that doesn't imply that reasoning involving those conditional probabilities will be invalid -- Bayes' theorem is true as a result of us reasoning using conditionals that condition on events that we know don't hold.

The conditional probability $P(B|A_j)$ is a perfectly valid conditional probability, whether or not $A_j$ actually obtains.

It's perfectly okay to reason via conditional probabilities that relate to conditions that don't hold; the results are logically valid. [Indeed, I bet you do it constantly without any concern.]

For example, if I say "Alison would have her umbrella if it were raining" and use this plus some data to support a conclusion: "She doesn't have her umbrella, so it's not raining", my conclusion doesn't become invalid because the conditional was untrue (The fact that "it's not raining" doesn't endanger the truth of the conditional that reasoning was based on: "if it were raining").

Glen_b
  • 257,508
  • 32
  • 553
  • 939
3

The principle is like a "fuzzy" version of the contraposition principle (or reductio ad absurdum principle, I'm not sure).

Consider that every dog has four legs. Then if you sample an animal with two legs you are sure it is not a dog.

Now only consider that every dog has a high probability to have four legs (in other words a high majority of dogs have four legs). Then if you sample an animal with two legs you conclude it is unlikely a dog. This is the principle of hypothesis testing (but in practice it requires a sensible choice of the event having high probability under $H_0$).

Stéphane Laurent
  • 17,425
  • 5
  • 59
  • 101
  • 1
    It's contraposition. In the philosophy of science, it is sometimes called the hypothetico-deductive model. – CloseToC Nov 16 '14 at 10:45