How to verify if data follows Zipf's law without looking at the graph

Question

I want to check if a given text sample was written by real people or not so I think Zipf's law could help.

If data follows Zipfian distribution, the most frequent word will occur approximately twice as often as the second most frequent word, three times as often as the third most frequent word, etc.

So that's the expected words distribution. A plot of frequency of word usage should look something like this:

For example, check http://www.gutenberg.org/files/50392/50392-0.txt, 140291 words, 10546 unique words, top frequencies are:

11538 7020 4765 3879 3375 2274 1522 1473 1240 1216 1005 1002 974 949 946 925 893 840 789 781

Not really match the frequency that Zipf’s law predicts. But we can look at the plot and know that it has a right curve so how we teach the machine to know that? And maybe when the data no longer follow Zipf's law, like top 3 frequencies all occur more than 10000 times, the machine will let us know?

Answer 1

You can't actually verify that your data does come from Zipf's law, but you may be able to tell that it doesn't. You can make some assessment of whether it's consistent with Zipf's law (at least to the degree that you can tell with the data you have).

The paper by Clauset, Shalizi and Newman[1] gives an explicit recipe to follow in Box 1 $^\dagger$ in their paper.

$^\dagger$ this is located at the top of the third page of the arXiv version of their paper (linked under the full paper reference below)

I believe it's brief enough to simply quote:

... In broad outline, however, the recipe we propose for the analysis of power-law data is straightforward and goes as follows.

1. Estimate the parameters $x_\text{min}$ and $α$ of the power-law model using the methods described in Section 3.

2. Calculate the goodness-of-fit between the data and the power law using the method described in Section 4. If the resulting p-value is greater than 0.1 the power law is a plausible hypothesis for the data, otherwise it is rejected.

3. Compare the power law with alternative hypotheses via a likelihood ratio test, as described in Section 5. For each alternative, if the calculated likelihood ratio is significantly different from zero, then its sign indicates whether the alternative is favored over the power-law model or not.

Step 3, the likelihood ratio test for alternative hypotheses, could in principle be replaced with any of several other established and statistically principled approaches for model comparison, such as a fully Bayesian approach [32], a cross-validation approach [59], or a minimum description length approach [20], although none of these methods are described here.

See:

[1]: Clauset A., C.R. Shalizi, and M. E. J. Newman (2009),
"Power-Law Distributions in Empirical Data,"
SIAM Rev., 51(4), 661–703. (43 pages)
http://epubs.siam.org/doi/abs/10.1137/070710111
(arXiv version)

(also see Shalizi's So you think you have a power law)

Ahem: Did you spot the error? Note that when they say "if the calculated likelihood ratio is significantly different from zero" they are actually referring to the log of the likelihood ratio.

[Disclaimer: I generally think that explicit hypothesis testing of goodness of fit answers the wrong question, and this case is not really an exception, but there are several aspects to the above paper that reduce my usual concerns somewhat. In any case it is very much worth reading, and contains a good deal of very sensible advice.]

Answer 2

Partial answers may reside in parametric tests. The following could be of interest:

• A simple and efficient test for Zipf’s law, Economics Letters, 2000,
• Testing for Zipf’s Law: A Common Pitfall, Economics Letters, 2011.