A psychology journal banned p-values and confidence intervals; is it indeed wise to stop using them?

Question

On 25 February 2015, the journal Basic and Applied Social Psychology issued an editorial banning $p$-values and confidence intervals from all future papers.

Specifically, they say (formatting and emphasis are mine):

• [...] prior to publication, authors will have to remove all vestiges of the NHSTP [null hypothesis significance testing procedure] ($p$-values, $t$-values, $F$-values, statements about ‘‘significant’’ differences or lack thereof, and so on).

• Analogous to how the NHSTP fails to provide the probability of the null hypothesis, which is needed to provide a strong case for rejecting it, confidence intervals do not provide a strong case for concluding that the population parameter of interest is likely to be within the stated interval. Therefore, confidence intervals also are banned from BASP.

• [...] with respect to Bayesian procedures, we reserve the right to make case-by-case judgments, and thus Bayesian procedures are neither required nor banned from BASP.

• [...] Are any inferential statistical procedures required? -- No [...] However, BASP will require strong descriptive statistics, including effect sizes.

Let us not discuss problems with and misuse of $p$-values here; there already are plenty of excellent discussions on CV that can be found by browsing the p-value tag. The critique of $p$-values often goes together with an advice to report confidence intervals for parameters of interest. For example, in this very well-argued answer @gung suggests to report effect sizes with confidence intervals around them. But this journal bans confidence intervals as well.

What are the advantages and disadvantages of such an approach to presenting data and experimental results as opposed to the "traditional" approach with $p$-values, confidence intervals, and significant/insignificant dichotomy? The reaction to this ban seems to be mostly negative; so what are the disadvantages then? American Statistical Association has even posted a brief discouraging comment on this ban, saying that "this policy may have its own negative consequences". What could these negative consequences be?

Or as @whuber suggested to put it, should this approach be advocated generally as a paradigm of quantitative research? And if not, why not?

PS. Note that my question is not about the ban itself; it is about the suggested approach. I am not asking about frequentist vs. Bayesian inference either. The Editorial is pretty negative about Bayesian methods too; so it is essentially about using statistics vs. not using statistics at all.

---

Other discussions: reddit, Gelman.

Answer 1

The first sentence of the current 2015 editorial to which the OP links, reads:

The Basic and Applied Social Psychology (BASP) 2014 Editorial *emphasized* that the null hypothesis significance testing procedure (NHSTP) is invalid...

(my emphasis)

In other words, for the editors it is an already proven scientific fact that "null hypothesis significance testing" is invalid, and the 2014 editorial only emphasized so, while the current 2015 editorial just implements this fact.

The misuse (even maliciously so) of NHSTP is indeed well discussed and documented. And it is not unheard of in human history that "things get banned" because it has been found that after all said and done, they were misused more than put to good use (but shouldn't we statistically test that?). It can be a "second-best" solution, to cut what on average (inferential statistics) has come to losses, rather than gains, and so we predict (inferential statistics) that it will be detrimental also in the future.

But the zeal revealed behind the wording of the above first sentence, makes this look -exactly, as a zealot approach rather than a cool-headed decision to cut the hand that tends to steal rather than offer. If one reads the one-year older editorial mentioned in the above quote (DOI:10.1080/01973533.2014.865505), one will see that this is only part of a re-hauling of the Journal's policies by a new Editor.

Scrolling down the editorial, they write

...On the contrary, we believe that the p<.05 bar is too easy to pass and sometimes serves as an excuse for lower quality research.

So it appears that their conclusion related to their discipline is that null-hypotheses are rejected "too-often", and so alleged findings may acquire spurious statistical significance. This is not the same argument as the "invalid" dictum in the first sentence.

So, to answer to the question, it is obvious that for the editors of the journal, their decision is not only wise but already late in being implemented: they appear to think that they cut out what part of statistics has become harmful, keeping the beneficial parts -they don't seem to believe that there is anything here that needs replacing with something "equivalent".

Epistemologically, this is an instance where scholars of a social science partially retract back from an attempt to make their discipline more objective in its methods and results by using quantitative methods, because they have arrived at the conclusion (how?) that, in the end, the attempt created "more bad than good". I would say that this is a very important matter, in principle possible to have happened, and one that would require years of work to demonstrate it "beyond reasonable doubt" and really help your discipline. But just one or two editorials and papers published will most probably (inferential statistics) just ignite a civil war.

The final sentence of the 2015 editorial reads:

We hope and anticipate that banning the NHSTP will have the effect of increasing the quality of submitted manuscripts by liberating authors from the stultified structure of NHSTP thinking thereby eliminating an important obstacle to creative thinking. The NHSTP has dominated psychology for decades; we hope that by instituting the first NHSTP ban, we demonstrate that psychology does not need the crutch of the NHSTP, and that other journals follow suit.

Answer 2

I feel that banning hypothesis tests is a great idea except for a select few "existence" hypotheses, e.g. testing the null hypothesis that there is not extra-sensory perception where all one would need to demonstrate to have evidence that ESP exists is non-randomness. But I think the journal missed the point that the main driver of poor research in psychology is the use of a threshold on $P$-values. It has been demonstrated in psychology and most other fields that a good deal of gaming goes on to arrive at $P < 0.05$. This includes hypothesis substitution, removing of observations, and subsetting data. It is thresholds that should be banned first.

The banning of confidence intervals is also overboard, but not for the reasons others have stated. Confidence intervals are useful only if one misinterprets them as Bayesian credible intervals (for suitable non-information priors). But they are still useful. The fact that their exact frequentist interpretation leads to nothing but confusion implies that we need to "get out of Dodge" and go Bayesian or likelihood school. But useful results can be obtained by misinterpreting good old confidence limits.

It is a shame that the editors of the journal misunderstood Bayesian statistics and don't know of the existence of pure likelihood inference. What they are seeking can be easily provided by Bayesian posterior distributions using slightly skeptical priors.

Answer 3

I see this approach as an attempt to address the inability of social psychology to replicate many previously published 'significant findings.'

Its disadvantages are:

1. that it doesn't address many of the factors leading to spurious effects. E.g.,

   • A) People can still peek at their data and stop running their studies when an effect size strikes them as being sufficiently large to be of interest.

   • B) Large effects sizes will still appear to have large power in retrospective assessments of power.

   • C) People will still fish for interesting and big effects (testing a bunch of hypotheses in an experiment and then reporting the one that popped up) or

   • D) pretend that an unexpected weird effect was expected all along.

   Shouldn't efforts be made to address these issues first?

2. As a field going forwards it will make a review of past findings pretty awful. There is no way to quantitatively assess the believability of different studies. If every journal implemented this approach, you'll have a bunch of social scientists saying there is evidence for X when it is totally unclear how believable X is and scientists arguing about how to interpret a published effect or arguing about whether it is important or worth talking about. Isn't this the point of having stats? To provide a consistent way to assess numbers. In my opinion, this new approach would cause a mess if it was widely implemented.

3. This change does not encourage researchers to submit the results of studies with small effect sizes so it doesn't really address the file-drawer effect (or are they going to publish findings with large n's regardless of effect size?). If we published all results of carefully designed studies, then even though the believability of results of individual studies may be uncertain, meta-analyses and reviews of studies that supplied statistical analysis would do a much better job at identifying the truth.

Answer 4

I came across a wonderful quote that almost argues for the same point, but not quite -- since it is an opening paragraph in a textbook that is mostly about frequentist statistics and hypothesis testing.

It is widely held by non-statisticians, like the author, that if you do good experiments statistics are not necessary. They are quite right. [...] The snag, of course, is that doing good experiments is difficult. Most people need all the help they can get to prevent them making fools of themselves by claiming that their favourite theory is substantiated by observations that do nothing of the sort. And the main function of that section of statistics that deals with tests of significance is to prevent people making fools of themselves. From this point of view, the function of significance tests is to prevent people publishing experiments, not to encourage them. Ideally, indeed, significance tests should never appear in print, having been used, if at all, in the preliminary stages to detect inadequate experiments, so that the final experiments are so clear that no justification is needed.

-- David Colquhoun, Lectures on biostatistics, 1971