Reviewing statistics in papers

Question

For some of us, refereeing papers is part of the job. When refereeing statistical methodology papers, I think advice from other subject areas is fairly useful, i.e. computer science and Maths.

This question concerns reviewing more applied statistical papers. By this I mean, the paper is submitted to a non-statistical/mathematical journal and statistics is just mentioned in the "methods" section.

Some particular questions:

1. How much effort should we put in to understand the application area?
2. How much time should I spend on a report?
3. How picky are you when looking at figures/tables.
4. How do you cope with the data not being available.
5. Do you try and rerun the analysis used.
6. What's the maximum number of papers your would review in a year?

Have a missed any questions? Feel free to edit or add a comment.

Edit

I coming to this question as a statistician reviewing a biology paper, but I'm interested in the statistical review of any non-mathematical discipline.

---

I'm not sure if this should be a CW. On one hand it's a bit open, but on the other I can see myself accepting an answer. Also, answers will probably be fairly long.

Answer 1

I am not sure about which area of science you are referring to (I'm sure the answer would be really different if dealing with biology vs physics for instance...)

Anyway, as a biologist, I will answer from a "biological" point of view:

How much effort should we put in to understand the application area?

I tend at least to read the previous papers from the same authors and look for a few review on the subject if I am not too familiar with it. This is especially true when dealing with new techniques I don't know, because I need to understand if they did all the proper controls etc.

How much time should I spend on a report?

As much as needed (OK, dumb answer, I know! :P) In general I would not like someone reviewing my paper to do an approximative job just because he/she has other things to do, so I try not to do it myself.

How picky are you when looking at figures/tables.

Quite picky. Figures are the first thing you look at when browsing through a paper. They need to be consistent (e.g. right titles on the axes, correct legend etc.). On occasion I have suggested to use a different kind of plot to show data when I thought the one used was not the best. This happens a lot in biology, a field that is dominated by the "barplot +/- SEM" type of graph. I'm also quite picky on the "materials and methods" section: a perfect statistical analysis on a inherently wrong biological model is completely useless.

How do you cope with the data not being available.

You just do and trust the Authors, I guess. In many cases in biology there's not much you can do, especially when dealing with things like imaging or animal behaviour and similar. Unless you want people to publish tons of images, videos etc (that you most likely would not go through anyways), but that may be very unpractical. If you think the data are really necessary ask for the authors to provide them as supplementary data/figures.

Do you try and rerun the analysis used.

Only if I have serious doubts on the conclusions drawn by the authors. In biology there's often a difference between what is (or not) "statistically significant" and what is "biologically significant". I prefer a thinner statistical analysis with good biological reasoning then the other way around. But again, in the very unlikely event that I were to review a bio-statistics paper (ahah, that would be some fun!!!) I would probably pay much more attention to the stats than to the biology in there.

Answer 2

This addresses the new question #6: "What's the maximum number of papers you would review in a year?" I'm responding as a member of several editorial boards. The perennial problem is finding enough reviewers. Depending on the journal, every submitted paper needs one to three peer reviewers, usually three. If the journal has an $x$% acceptance rate, then the mean number of reviews per accepted paper obviously is around $3/(x/100)$. E.g., if the acceptance rate is 33%, the editors need to obtain nine reviews for every paper published. If you, as an author, take this seriously, then you should attempt to provide nine reviews (or whatever the number turns out to be for your target journals) for every paper you publish!

I was moved to write this due to the strong parallel with voting on this site: in order for you to garner a reputation of $r$, other people have to upvote some combination of $r/10$ of your answers and $r/5$ of your questions. Thus, if you're pulling your weight, a check of your profile should show at least $r/10$ upvotes. That is the case for many but certainly not all of the most active members of this site. Something to think about... Remember to vote!

Answer 3

My POV would be reviewing a paper in psychology or forecasting on its statistical merits. I'll mostly second Nico's very good remarks.

How much effort should we put in to understand the application area?

Quite a lot, actually. I wouldn't trust myself to comment on more than the most basic statistical problems without having understood the area. Fortunately, this is often not very hard in many branches of psychology.

How much time should I spend on a report?

I'll go out on a limb and state a specific time: I'll spend anything between two and eight hours on a review, sometimes more. If I find that I'm spending more than a day on a paper, it probably means that I'm really not qualified to understand it, so I'll recommend the journal find someone else (and try to suggest some people).

How picky are you when looking at figures/tables.

Very picky indeed. The figures are going to be what people remember of a paper and what ends up in lecture presentations without much context, so these really need to be done well.

How do you cope with the data not being available.

In psychology, the data are usually not shared - measuring 50 people by MRI is very expensive, and the authors will want to use these data for further papers, so I kind of understand their reluctance to just give out the data. So anyone who does share their data gets a big bonus in my book, but not sharing is understandable.

In forecasting, many datasets are publicly available. In this case I usually recommend that the authors share their code (and do so myself).

Do you try and rerun the analysis used.

Without the data, there is only so much one can learn from this. I'll play around with simulated data if something is very surprising about the paper's results; otherwise one can often tell appropriate from inappropriate methods without the data (once one understands the area, see above).

What's the maximum number of papers your would review in a year?

There is really little to add to whuber's point above - assuming that every paper with on average n coauthors I (co-)submit gets 3 reviews, one should really aim at reviewing at least 3/(n+1) papers for each own submission (counting submissions rather than own papers which may be rejected and resubmitted). And of course, the number of submissions as well as the number of coauthors varies strongly with the discipline.