Is the "hybrid" between Fisher and Neyman-Pearson approaches to statistical testing really an "incoherent mishmash"?

Question

There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these two approaches, the claim goes, are "incompatible" and hence the resulting "hybrid" is an "incoherent mishmash". I will provide a bibliography and some quotes below, but for now suffice it to say that there is a lot written about that in the wikipedia article on Statistical hypothesis testing. Here on CV, this point was repeatedly made by @Michael Lew (see here and here).

My question is: why are F and N-P approaches claimed to be incompatible and why is the hybrid claimed to be incoherent? Note that I read at least six anti-hybrid papers (see below), but still fail to understand the problem or the argument. Note also, that I am not suggesting to debate if F or N-P is a better approach; neither am I offering to discuss frequentist vs. Bayesian frameworks. Instead, the question is: accepting that both F and N-P are valid and meaningful approaches, what is so bad about their hybrid?

---

Here is how I understand the situation. Fisher's approach is to compute the $p$-value and take it as an evidence against the null hypothesis. The smaller the $p$, the more convincing the evidence. The researcher is supposed to combine this evidence with his background knowledge, decide if it is convincing enough, and proceed accordingly. (Note that Fisher's views changed over the years, but this is what he seems to have eventually converged to.) In contrast, Neyman-Pearson approach is to choose $\alpha$ ahead of time and then to check if $p\le\alpha$; if so, call it significant and reject the null hypothesis (here I omit large part of the N-P story that has no relevance for the current discussion). See also an excellent reply by @gung in When to use Fisher and Neyman-Pearson framework?

The hybrid approach is to compute the $p$-value, report it (implicitly assuming that the smaller the better), and also call the results significant if $p\le\alpha$ (usually $\alpha=0.05$) and nonsignificant otherwise. This is supposed to be incoherent. How can it be invalid to do two valid things simultaneously, beats me.

As particularly incoherent the anti-hybridists view the widespread practice of reporting $p$-values as $p<0.05$, $p<0.01$, or $p<0.001$ (or even $p\ll0.0001$), where always the strongest inequality is chosen. The argument seems to be that (a) the strength of evidence cannot be properly assessed as exact $p$ is not reported, and (b) people tend to interpret the right-hand number in the inequality as $\alpha$ and view it as type I error rate, and that is wrong. I fail to see a big problem here. First, reporting exact $p$ is certainly a better practice, but nobody really cares if $p$ is e.g. $0.02$ or $0.03$, so rounding it on a log scale is not soooo bad (and going below $\sim 0.0001$ does not make sense anyway, see How should tiny p-values be reported?). Second, if the consensus is to call everything below $0.05$ significant, then error rate will be $\alpha=0.05$ and $p \ne \alpha$, as @gung explains in Interpretation of p-value in hypothesis testing. Even though this is potentially a confusing issue, it does not strike me as more confusing than other issues in statistical testing (outside of the hybrid). Also, every reader can have her own favourite $\alpha$ in mind when reading a hybrid paper, and her own error rate as a consequence. So what is the big deal?

One of the reasons I want to ask this question is because it literally hurts to see how much of the wikipedia article on Statistical hypothesis testing is devoted to lambasting hybrid. Following Halpin & Stam, it claims that a a certain Lindquist is to blame (there is even a large scan of his textbook with "errors" highlighted in yellow), and of course the wiki article about Lindquist himself starts with the same accusation. But then, maybe I am missing something.

---

References

• Gigerenzer, 1993, The superego, the ego, and the id in statistical reasoning -- introduced the term "hybrid" and called it "incoherent mishmash"

  • See also more recent expositions by Gigerenzer et al.: e.g. Mindless statistics (2004) and The Null Ritual. What You Always Wanted to Know About Significance Testing but Were Afraid to Ask (2004).

• Cohen, 1994, The Earth Is Round ($p<.05$) -- a very popular paper with almost 3k citations, mostly about different issues but favourably citing Gigerenzer

• Goodman, 1999, Toward evidence-based medical statistics. 1: The P value fallacy

• Hubbard & Bayarri, 2003, Confusion over measures of evidence ($p$'s) versus errors ($\alpha$'s) in classical statistical testing -- one of the more eloquent papers arguing against "hybrid"

• Halpin & Stam, 2006, Inductive Inference or Inductive Behavior: Fisher and Neyman-Pearson Approaches to Statistical Testing in Psychological Research (1940-1960) [free after registration] -- blames Lindquist's 1940 textbook for introducing the "hybrid" approach

• @Michael Lew, 2006, Bad statistical practice in pharmacology (and other basic biomedical disciplines): you probably don't know P -- a nice review and overview

Quotes

Gigerenzer: What has become institutionalized as inferential statistics in psychology is not Fisherian statistics. It is an incoherent mishmash of some of Fisher's ideas on one hand, and some of the ideas of Neyman and E. S. Pearson on the other. I refer to this blend as the "hybrid logic" of statistical inference.

Goodman: The [Neyman-Pearson] hypothesis test approach offered scientists a Faustian bargain -- a seemingly automatic way to limit the number of mistaken conclusions in the long run, but only by abandoning the ability to measure evidence [a la Fisher] and assess truth from a single experiment.

Hubbard & Bayarri: Classical statistical testing is an anonymous hybrid of the competing and frequently contradictory approaches [...]. In particular, there is a widespread failure to appreciate the incompatibility of Fisher's evidential $p$ value with the Type I error rate, $\alpha$, of Neyman-Pearson statistical orthodoxy. [...] As a prime example of the bewilderment arising from [this] mixing [...], consider the widely unappreciated fact that the former's $p$ value is incompatible with the Neyman-Pearson hypothesis test in which it has become embedded. [...] For example, Gibbons and Pratt [...] erroneously stated: "Reporting a P-value, whether exact or within an interval, in effect permits each individual to choose his own level of significance as the maximum tolerable probability of a Type I error."

Halpin & Stam: Lindquist's 1940 text was an original source of the hybridization of the Fisher and Neyman-Pearson approaches. [...] rather than adhering to any particular interpretation of statistical testing, psychologists have remained ambivalent about, and indeed largely unaware of, the conceptual difficulties implicated by the Fisher and Neyman-Pearson controversy.

Lew: What we have is a hybrid approach that neither controls error rates nor allows assessment of the strength of evidence.

Answer 1

I believe the papers, articles, posts e.t.c. that you diligently gathered, contain enough information and analysis as to where and why the two approaches differ. But being different does not mean being incompatible.

The problem with the "hybrid" is that it is a hybrid and not a synthesis, and this is why it is treated by many as a hybris, if you excuse the word-play.
Not being a synthesis, it does not attempt to combine the differences of the two approaches, and either create one unified and internally consistent approach, or keep both approaches in the scientific arsenal as complementary alternatives, in order to deal more effectively with the very complex world we try to analyze through Statistics (thankfully, this last thing is what appears to be happening with the other great civil war of the field, the frequentist-bayesian one).

The dissatisfaction with it I believe comes from the fact that it has indeed created misunderstandings in applying the statistical tools and interpreting the statistical results, mainly by scientists that are not statisticians, misunderstandings that can have possibly very serious and damaging effects (thinking about the field of medicine helps giving the issue its appropriate dramatic tone). This misapplication, is I believe, accepted widely as a fact-and in that sense, the "anti-hybrid" point of view can be considered as widespread (at least due to the consequences it had, if not for its methodological issues).

I see the evolution of the matter so far as a historical accident (but I don't have a $p$-value or a rejection region for my hypothesis), due to the unfortunate battle between the founders. Fisher and Neyman/Pearson have fought bitterly and publicly for decades over their approaches. This created the impression that here is a dichotomous matter: the one approach must be "right", and the other must be "wrong".

The hybrid emerged, I believe, out of the realization that no such easy answer existed, and that there were real-world phenomena to which the one approach is better suited than the other (see this post for such an example, according to me at least, where the Fisherian approach seems more suitable). But instead of keeping the two "separate and ready to act", they were rather superfluously patched together.

I offer a source which summarizes this "complementary alternative" approach: Spanos, A. (1999). Probability theory and statistical inference: econometric modeling with observational data. Cambridge University Press., ch. 14, especially Section 14.5, where after presenting formally and distinctly the two approaches, the author is in a position to point to their differences clearly, and also argue that they can be seen as complementary alternatives.

Answer 2

My own take on my question is that there is nothing particularly incoherent in the hybrid (i.e. accepted) approach. But as I was not sure if I am maybe failing to comprehend the validity of the arguments presented in the anti-hybrid papers, I was happy to find the discussion published together with this paper:

• Hubbard & Bayarri, 2003, Confusion over measures of evidence (p's) versus errors (α's) in classical statistical testing

Unfortunately, two replies published as a discussion were not formatted as separate articles and so cannot be properly cited. Still, I would like to quote from both of them:

Berk: The theme of Sections 2 and 3 seems to be that Fisher did not like what Neyman and Pearson did, and Neyman did not like what Fisher did, and therefore we should not do anything that combines the two approaches. There is no escaping the premise here, but the reasoning escapes me.

Carlton: the authors adamantly insist that most confusion stems from the marriage of Fisherian and Neyman-Pearsonian ideas, that such a marriage is a catastrophic error on the part of modern statisticians [...] [T]hey seem intent on establishing that P values and Type I errors cannot coexist in the same universe. It is unclear whether the authors have given any substantive reason why we cannot utter "p value" and "Type I error" in the same sentence. [...] The "fact" of their [F and NP] incompatibility comes as surprising news to me, as I'm sure it does to the thousands of qualified statisticians reading the article. The authors even seem to suggest that among the reasons statisticians should now divorce these two ideas is that Fisher and Neyman were not terribly fond of each other (or each other's philosophies on testing). I have always viewed our current practice, which integrates Fisher's and Neyman's philosophies and permits discussion of both P values and Type I errors -- though certainly not in parallel -- as one of our discipline's greater triumphs.

Both responses are very worth reading. There is also a rejoinder by the original authors, which does not sound convincing to me at all.

Answer 3

An often seen (and supposedly accepted) union (or better: "hybrid") between the two approaches is as follows:

1. Set a prespecified level $\alpha$ (0.05 say)
2. Then test your hypothesis, e.g. $H_o: \mu = 0$ vs. $H_1: \mu \ne 0$

3. State the p value and formulate your decision based on the level $\alpha$:

   If the resulting p value is below $\alpha$, you could say

   • "I reject $H_o$" or
   • "I reject $H_o$" in favor of $H_1$" or
   • "I am $100\% \cdot (1-\alpha)$ certain that $H_1$ holds"

   If the p value is not small enough, you would say

   • "I cannot reject $H_o$" or
   • "I cannot reject $H_o$ in favor of $H_1$"

Here, aspects from Neyman-Pearson are:

• You decide something
• You have an alternative hypothesis at hand (although it is just the contrary of $H_o$)
• You know the type I error rate

Fisherian aspects are:

• You state the p value. Any reader has thus the possibility to use its own level (e.g. strictly correcting for multiple testing) for decision
• Basically, only the null hypothesis is required since the alternative is just the contrary
• You don't know the type II error rate. (But you could immediately get it for specific values of $\mu \ne 0$.)

ADD-ON

While it is good to be aware of the discussion about the philosophical problems of Fisher's, NP's or this hybrid approach (as taught in almost religious frenzy by some), there are much more relevant issues in statistics to fight against:

• Asking uninformative questions (like binary yes/no questions instead of quantitative "how much" questions, i.e. using tests instead of confidence intervals)
• Data driven analysis methods that lead to biased results (stepwise regression, testing assumptions etc.)
• Choosing wrong tests or methods
• Misinterpreting results
• Using classic statistics for non-random samples

Answer 4

I fear that a real response to this excellent question would require a full-length paper. However, here are a couple of points that are not present in either the question or the current answers.

1. The error rate 'belongs' to the procedure but the evidence 'belongs' to the experimental results. Thus it is possible with multi-stage procedures with sequential stopping rules to have a result with very strong evidence against the null hypothesis but a not significant hypothesis test result. That can be thought of as a strong incompatibility.

2. If you are interested in the incompatibilities, you should be interested in the underlying philosophies. The philosophical difficulty comes from a choice between compliance with the Likelihood Principle and compliance with the Repeated Sampling Principle. The LP says roughly that, given a statistical model, the evidence in a dataset relevant to the parameter of interest is completely contained in the relevant likelihood function. The RSP says that one should prefer tests that give error rates in the long run that equal their nominal values.

Answer 5

accepting that both F and N-P are valid and meaningful approaches, what is so bad about their hybrid?

Short answer: the use of a nil (no difference, no correlation) null hypothesis irregardless of the context. Everything else is a "misuse" by people who have created myths for themselves about what the process can achieve. The myths arise from people attempting to reconcile their (sometimes appropriate) use of trust in authority and consensus heuristics with the inapplicability of the procedure to their problem.

As far as I know Gerd Gigerenzer came up with term "hybrid":

I asked the author [a distinguished statistical textbook author, whose book went through many editions, and whose name does not matter] why he removed the chapter on Bayes as well as the innocent sentence from all subsequent editions. “What made you present statistics as if it had only a single hammer, rather than a toolbox? Why did you mix Fisher’s and Neyman–Pearson’s theories into an inconsistent hybrid that every decent statistician would reject?”

To his credit, I should say that the author did not attempt to deny that he had produced the illusion that there is only one tool. But he let me know who was to blame for this. There were three culprits: his fellow researchers, the university administration, and his publisher. Most researchers, he argued, are not really interested in statistical thinking, but only in how to get their papers published [...]

The null ritual:

1. Set up a statistical null hypothesis of “no mean difference” or “zero correlation.” Don’t specify the predictions of your research hypothesis or of any alternative substantive hypotheses.

2. Use 5% as a convention for rejecting the null. If significant, accept your research hypothesis. Report the result as $p < 0.05$, $p < 0.01$ , or $p < 0.001$ (whichever comes next to the obtained $p$-value).

3. Always perform this procedure.

Gigerenzer, G (November 2004). "Mindless statistics". The Journal of Socio-Economics 33 (5): 587–606. doi:10.1016/j.socec.2004.09.033.

Edit: And we should always need to mention, because the "hybrid" is so slippery and ill-defined, that using the nil null to get a p-value is perfectly fine as a way to compare effect sizes given different sample sizes. It is the "test" aspect that introduces the problem.

Edit 2: @amoeba A p-value can be fine as a summary statistic, in this case the nil null hypothesis is just an arbitrary landmark: http://arxiv.org/abs/1311.0081. However, as soon as you start trying to draw a conclusion or make a decision (ie "test" the null hypothesis) it stops making sense. In the comparing two groups example, we want to know how different two groups are and the various possible explanations there may be for differences of that magnitude and type.

The p value can be used as a summary statistic telling us the magnitude of the difference. However, using it to "disprove/reject" zero difference serves no purpose that I can tell. Also, I think many of these study designs that compare average measurements of living things at a single timepoint are misguided. We should want to observe how individual instances of the system change over time, then come up with a process that explains the pattern observed (including any group differences).

Answer 6

I see that those with more expertise than myself have provided answers, but I think my answer has the potential to add something additional, so I'll offer this as one other layman's perspective.

Is the hybrid approach incoherent?  I'd say it depends on whether or not the researcher ends up acting inconsistently with the rules that they started out with: specifically the yes/no rule that comes into play with the setting of an alpha value.

Incoherent

Start with Neyman-Pearson.  Researcher sets alpha=0.05, runs the experiment, calculates p=0.052.  Researcher looks at that p-value and, using Fisherian inference (often implicitly), considers the result to be sufficiently incompatible with the test hypothesis that they will still claim "something" is going on.  The result is somehow "good enough" even though the p-value was greater than the alpha value.  Often this is paired with language such as "nearly significant" or "trending towards significance" or some wording along those lines.

However, setting an alpha value before running the experiment means that one has chosen the approach of Neyman-Pearson inductive behavior.  Choosing to ignore that alpha value after calculating the p-value, and thus claiming something is still somehow interesting, undermines the entire approach that one started with.  If a researcher starts down Path A (Neyman-Pearson), but then jumps across to another path (Fisher) once they don't like the path they are on, I consider that incoherent.  They are not being consistent with the (implied) rules that they started with.

Coherent (possibly)

Start with N-P.  Researcher sets alpha=0.05, runs the experiment, calculates p=0.0014.  Researcher observes that p < alpha, and thus rejects the test hypothesis (typically no effect null) and accepts the alternative hypothesis (the effect is real).  At this point the researcher, in addition to deciding to treat the outcome as a real effect (N-P), decides to infer (Fisher) that the experiment provides very strong evidence that the effect is real.  They have added nuance to the approach they started with, but have not contradicted the rules set in place by choosing an alpha value at the beginning.

Summary

If one starts by choosing an alpha value, then one has decided to take the Neyman-Pearson path and follow the rules for that approach.  If they, at some point, violate those rules using Fisherian inference as the justification, then they have acted inconsistently/incoherently.

I suppose one could go a step further and declare that because it is possible to use the hybrid incoherently, therefore the approach is inherently incoherent, but that seems to be getting deeper into the philosophical aspects, which I don't consider myself qualified to even offer an opinion on.

Hat tip to Michael Lew.  His 2006 article helped me understand these issues better than any other resource.