Statistical forensics: Benford and beyond

Question

What broad methods are there to detect fraud, anomalies, fudging, etc. in scientific works produced by a third party? (I was motivated to ask this by the recent Marc Hauser affair.) Usually for election and accounting fraud, some variant of Benford's Law is cited. I am not sure how this could be applied to e.g. the Marc Hauser case, because Benford's Law requires numbers to be approximately log uniform.

As a concrete example, suppose a paper cited the p-values for a large number of statistical tests. Could one transform these to log uniformity, then apply Benford's Law? It seems like there would be all kinds of problems with this approach (e.g. some of the null hypotheses might legitimately be false, the statistical code might give p-values which are only approximately correct, the tests might only give p-values which are uniform under the null asymptotically, etc.)

Answer 1

Great Question!

In the scientific context there are various kinds of problematic reporting and problematic behaviour:

• Fraud: I'd define fraud as a deliberate intention on the part of the author or analyst to misrepresent the results and where the misrepresentation is of a sufficiently grave nature. The main example being complete fabrication of raw data or summary statistics.
• Error: Data analysts can make errors at many phases of data analysis from data entry, to data manipulation, to analyses, to reporting, to interpretation.
• Inappropriate behaviour: There are many forms of inappropriate behaviour. In general, it can be summarised by an orientation which seeks to confirm a particular position rather than search for the truth.

Common examples of inappropriate behaviour include:

• Examining a series of possible dependent variables and only reporting the one that is statistically significant
• Not mentioning important violations of assumptions
• Performing data manipulations and outlier removal procedures without mentioning it, particularly where these procedures are both inappropriate and chosen purely to make the results look better
• Presenting a model as confirmatory which is actually exploratory
• Omitting important results that go against the desired argument
• Choosing a statistical test solely on the basis that it makes the results look better
• Running a series of five or ten under-powered studies where only one is statistically significant (perhaps at p = .04) and then reporting the study without mention of the other studies

In general, I'd hypothesise that incompetence is related to all three forms of problematic behaviour. A researcher who does not understand how to do good science but otherwise wants to be successful will have a greater incentive to misrepresent their results, and is less likely to respect the principles of ethical data analysis.

The above distinctions have implications for detection of problematic behaviour. For example, if you manage to discern that a set of reported results are wrong, it still needs to be ascertained as to whether the results arose from fraud, error or inappropriate behaviour. Also, I'd assume that various forms of inappropriate behaviour are far more common than fraud.

With regards to detecting problematic behaviour, I think it is largely a skill that comes from experience working with data, working with a topic, and working with researchers. All of these experiences strengthen your expectations about what data should look like. Thus, major deviations from expectations start the process of searching for an explanation. Experience with researchers gives you a sense of the kinds of inappropriate behaviour which are more or less common. In combination this leads to the generation of hypotheses. For example, if I read a journal article and I am surprised with the results, the study is underpowered, and the nature of the writing suggests that the author is set on making a point, I generate the hypothesis that the results perhaps should not be trusted.

Other Resources

• Robert P. Abelson Statistics as a Principled Argument has a chapter titled "On Suspecting Fishiness"

Answer 2

Actually, Benford's Law is an incredibly powerful method. This is because the Benford's frequency distribution of first digit is applicable to all sorts of data set that occur in the real or natural world.

You are right that you can use Benford's Law in only certain circumstances. You say that the data has to have a uniform log distribution. Technically, this is absolutely correct. But, you could describe the requirement in a much simpler and lenient way. All you need is that the data set range crosses at least one order of magnitude. Let's say from 1 to 9 or 10 to 99 or 100 to 999. If it crosses two orders of magnitude, you are in business. And, Benford's Law should be pretty helpful.

The beauty of Benford's Law is that it helps you narrow your investigation really quickly on the needle(s) within the hay stack of data. You look for the anomalies whereby the frequency of first digit is much different than Benford frequencies. Once you notice that there are two many 6s, you then use Benford's Law to focus on just the 6s; but, you take it now to the first two digits (60, 61, 62, 63, etc...). Now, maybe you find out there are a lot more 63s then what Benford suggest (you would do that by calculating Benford's frequency: log(1+1/63) that gives you a value close to 0%). So, you use Benford to the first three digits. By the time you find out there are way too many 632s (or whatever by calculating Benford's frequency: log (1+1/632)) than expected you are probably on to something. Not all anomalies are frauds. But, most frauds are anomalies.

If the data set that Marc Hauser manipulated are natural unconstrained data with a related range that was wide enough, then Benford's Law would be a pretty good diagnostic tool. I am sure there are other good diagnostic tools also detecting unlikely patterns and by combining them with Benford's Law you could most probably have investigated the Marc Hauser affair effectively (taking into consideration the mentioned data requirement of Benford's Law).

I explain Benford's Law a bit more in this short presentation that you can see here: http://www.slideshare.net/gaetanlion/benfords-law-4669483