Statistics published in academic papers

Question

I read a lot of evolutionary/ecological academic papers, sometimes with the specific aim of seeing how statistics are being used 'in the real world' outside of the textbook. I normally take the statistics in papers as gospel and use the papers to help in my statistical learning. After all, if a paper has taken years to write and has gone through rigorous peer review, then surely the statistics are going to be rock solid? But in the past few days, I've questioned my assumption, and wondered how often the statistical analysis published in academic papers is suspect? In particular, it might be expected that those in fields such as ecology and evolution have spent less time learning statistics and more time learning their fields.

How often do people find suspect statistics in academic papers?

Answer 1

After all, if a paper has taken years to write and has gone through rigorous peer review, then surely the statistics are going to be rock solid?

My experience of reading papers that attempt to apply statistics across a wide variety of areas (political science, economics, psychology, medicine, biology, finance, actuarial science, accounting, optics, astronomy, and many, many others) is that the quality of the statistical analysis may be anywhere on the spectrum from excellent and well done to egregious nonsense. I have seen good analysis in every one of the areas I have mentioned, and pretty poorly done analysis in almost all of them.

Some journals are generally pretty good, and some can be more like playing darts with a blindfold - you might get most of them not too terribly far off the target, but there's going to be a few in the wall, the floor and the ceiling. And maybe the cat.

I don't plan on naming any culprits, but I will say I have seen academic careers built on faulty use of statistics (i.e. where the same mistakes and misunderstandings were repeated in paper after paper, over more than a decade).

So my advice is let the reader beware; don't trust that the editors and peer reviewers know what they're doing. Over time you may get a good sense of which authors can generally be relied on to not do anything too shocking, and which ones should be treated especially warily. You may get a sense that some journals typically have very high standard for their stats.

But even a typically good author can make a mistake, or referees and editors can fail to pick up errors they might normally find; a typically good journal can publish a howler.

[Sometimes, you'll even see really bad papers win prizes or awards... which doesn't say much for the quality of the people judging the prize, either.]

I wouldn't like to guess what the fraction of "bad" stats I might have seen (in various guises, and at every stage from defining the question, design of the study, data collection, data management, ... right through to analysis and conclusions), but it's not nearly small enough for me to feel comfortable.

I could point to examples, but I don't think this is the right forum to do that. (It would be nice if there was a good forum for that, actually, but then again, it would likely become highly "politicized" quite quickly, and soon fail to serve its purpose.)

I've spent some time trawling through PLOS ONE ... and again, not going to point at specific papers. Some things I noticed: it looks like a large proportion of papers have stats in them, probably more than half having hypothesis tests. The main dangers seem to be lots of tests, either with high $\alpha$ like 0.05 on each (which is not automatically a problem as long as we understand that quite a few really tiny effects might show up as significant by chance), or an incredibly low individual significance level, which will tend to give low power. I also saw a number of cases where about half a dozen different tests were apparently applied to resolving exactly the same question. This strikes me as a generally bad idea. Overall the standard was pretty good across a few dozen papers, but in the past I have seen an absolutely terrible paper there.

[Perhaps I could indulge in just one example, indirectly. This question asks about one doing something quite dubious. It's far from the worst thing I've seen.]

On the other hand, I also see (even more frequently) cases where people are forced to jump through all kinds of unnecessary hoops to get their analysis accepted; perfectly reasonable things to do are not accepted because there's a "right" way to do things according to a reviewer or an editor or a supervisor, or just in the unspoken culture of a particular area.

Answer 2

I respect @Glen_b's stance on the right way to answer here (and certainly don't intend to detract from it), but I can't quite resist pointing to a particularly entertaining example that's close to my home. At the risk of politicizing things and doing this question's purpose a disservice, I recommend Wagenmakers, Wetzels, Boorsboom, and Van Der Maas ^{₍₂₀₁₁₎}. I cited this in a related post on the Cognitive Sciences beta SE (How does cognitive science explain distant intentionality and brain function in recipients?), which considers another example of "a dart hitting the cat". Wagenmakers and colleagues' article comments directly on a real "howler" though: it was published in JPSP (one of the biggest journals in psychology) a few years ago. They also argue more generally in favor of Bayesian analysis and that:

In order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal.

I probably don't need to tell you that this didn't exactly come across as preaching to the choir. FWIW, there is a rebuttal as well (as there always seems to be between Bayesians and frequentists; ^{_{(Bem, Utts, & Johnson, 2011)}}, but I get the feeling that it didn't exactly checkmate the debate.

Psychology as a scientific community has been on a bit of a replication kick recently, partly due to this and other high-profile methodological shortcomings. Other comments here point to cases similar to what were once known as voodoo correlations in social neuroscience (how's that for politically incorrect BTW? the paper has been retitled; ^{_{Vul, Harris, Winkielman, & Pashler, 2009}}). That too attracted its rebuttal, which you can check out for more debate of highly debatable practices.

For even more edutainment at the (more depersonalized) expense of (pseudo)statisticians behaving badly, see our currently 8th-most-upvoted question here on CV with another (admittedly) politically incorrect title, "What are common statistical sins?" Its OP @MikeLawrence attributes his inspiration to his parallel study of psychology and statistics. It's one of my personal favorites, and its answers are very useful for avoiding the innumerable pitfalls out there yourself.

---

On the personal side, I've been spending much of my last five months here largely because it's amazingly difficult to get rock-solid statistics on certain data-analytic questions. Frankly, peer review is often not very rigorous at all, especially in terms of statistical scrutiny of research in younger sciences with complex questions and plenty of epistemic complications. Hence I've felt the need to take personal responsibility for polishing the methods in my own work.

While presenting my dissertation research, I got a sense of how important personal responsibility for statistical scrutiny is. Two exceptional psychologists at my alma mater interjected that I was committing one of the most basic sins in my interpretations of correlations. I'd thought myself above it, and had lectured undergrads about it several times already, but I still went there, and got called out on it (early on, thank heavens). I went there because research I was reviewing and replicating went there! Thus I ended up adding several sections to my dissertation that called out those other researchers for assuming causality from quasi-experimental longitudinal studies (sometimes even from cross-sectional correlations) and ignoring alternative explanations prematurely.

My dissertation was accepted without revisions by my committee, which included another exceptional psychometrician and the soon-to-be-president of SPSP (which publishes JPSP), but to be frank once more, I'm not bragging in saying this. I've since managed to poke several rabbit holes in my own methods despite passing the external review process with perfectly good reviewers. I've now fallen into the deep end of stats in trying to plug them with methods more appropriate for predictive modeling of Likert ratings like SEM, IRT, and nonparametric analysis (see Regression testing after dimension reduction). I'm opting voluntarily to spend years on a paper that I could probably just publish as-is instead...I think I even have a simulation study left to do before I can proceed conscientiously.

Yet I emphasize that this is optional – maybe even overzealous and a costly luxury amidst the publish-or-perish culture that often emphasizes quantity over quality in early-career work records. Misapplication of parametric models for continuous data to assumption-violating distributions of ordinal data is all too common in my field, as is the misinterpretation and misrepresentation of statistical significance (see Accommodating entrenched views of p-values). I could totally get away with it (in the short term)...and it's not even all that hard to do better than that. I suppose I have several recent years of amazing advances in R programs to thank for that though! Here's hoping the times are changing.

---

^{References
· Bem, D. J., Utts, J., & Johnson, W. O. (2011). Must psychologists change the way they analyze their data? Journal of Personality and Social Psychology, 101(4), 716–719. Retrieved from http://deanradin.com/evidence/Bem2011.pdf.
· Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science, 4(3), 274–290. Retrieved from http://www.edvul.com/pdf/VulHarrisWinkielmanPashler-PPS-2009.pdf.
· Wagenmakers, E. J., Wetzels, R., Borsboom, D., & Van der Maas, H. (2011). Why psychologists must change the way they analyze their data: The case of psi. Journal of Personality and Social Psychology, 100, 426–432. Retrieved from http://mpdc.mae.cornell.edu/Courses/MAE714/Papers/Bem6.pdf.}

Answer 3

I recall at University being ask by a few final year social science students on different occasions (one of them got a 1st) how to work out an average for their project that had had a handful of data points. (So they were not having problem with using software, just with the concept of how to do the maths with a calculator.)

They just give me blank looks when I ask them what type of average they wanted.

Yet they all felt a need to put some stats in their report, as it was the done thing – I expect they have all read 101 papers that had stats without thinking about what the stats meant if anything.

It is clear that the researcher that taught them over the 3 years did not care about the correctness of stats enough to distil any understanding into the students.

(I was a computer Sci student at the time. I am posting this as an answer as it is a bit long for a comment.)

Answer 4

As a woefully incomplete list, I find statistics most frequently correct in 1) physics papers followed by 2) statistical papers and most miserable in 3) medical papers. The reasons for this are straightforward and have to do with the completeness of the requirements imposed upon the prototypical model in each field.

In physics papers, equations and applied statistics have to pay attention to balanced units and have the most frequent occurrence of causal relationships, and testing against physical standards.

In statistics, 1) units and causality are sometimes ignored, the assumptions are sometimes heuristic, and physical testing is too often ignored, but equality (or inequality), i.e., logic is generally preserved along an inductive path, where the latter cannot correct for unphysical assumptions.

In medicine, typically units are ignored, the equations and assumptions are typically heuristic, typically untested and frequently spurious.

Naturally, a field like statistical mechanics is more likely to have testable assumptions than, let us say, economics, and, that does not reflect on the talents of the prospective authors in those fields. It is more related to how much of what is being done is actually testable, and how much testing has been done historically in each field.

Answer 5

Any paper that disproves the nil null hypothesis is using worthless statistics (the vast majority of what I have seen). This process can provide no information not already provided by the effect size. Further it tells us nothing about whether a significant result is actually due to the cause theorized by the researcher. This requires thoughtful investigation of the data for evidence of confounds. Most often, if present, the strongest of this evidence is even thrown away as "outliers".

I am not so familiar with evolution/ecology, but in the case of psych and medical research I would call the level of statistical understanding "severely confused" and "an obstacle to scientific progress". People are supposed to be disproving something predicted by their theory, not the opposite of it (zero difference/effect).

There have been thousands of papers written on this topic. Look up NHST hybrid controversy.

Edit: And I do mean the nill null hypothesis significance test has a maximum of zero scientific value. This person hits the nail on the head:

http://www.johnmyleswhite.com/notebook/2012/05/18/criticism-4-of-nhst-no-mechanism-for-producing-substantive-cumulative-knowledge/

Also: Paul Meehl. 1967. Theory Testing in Psychology and Physics: A Methodological Paradox

Edit 3:

If someone has arguments in favor of the usefulness of strawman NHST that do not require thinking "reject the hypothesis that the rate of warming is the same, but DO NOT take this to imply that the rate of warming is the not same" is a rational statement, I would welcome your comments.

Edit 4:

What did Fisher mean by the following quote? Does it suggest that he thought "If model/theory A is incompatible with the data, we can say A is false, but nothing about whether not A is true"?

"it is certain that the interest of statistical tests for scientific workers depends entirely from their use in rejecting hypotheses which are thereby judged to be incompatible with the observations."

...

It would, therefore, add greatly to the clarity with which the tests of significance are regarded if it were generally understood that tests of significance, when used accurately, are capable of rejecting or invalidating hypotheses, in so far as these are contradicted by the data; but that they are never capable of establishing them as certainly true

Karl Pearson and R. A. Fisher on Statistical Tests: A 1935 Exchange from Nature

Is it that he assumed people would only try to invalidate plausible hypotheses rather than strawmen? Or am I wrong?