How does one measure the non-uniformity of a distribution?

Question

I'm trying to come up with a metric for measuring non-uniformity of a distribution for an experiment I'm running. I have a random variable that should be uniformly distributed in most cases, and I'd like to be able to identify (and possibly measure the degree of) examples of data sets where the variable is not uniformly distributed within some margin.

An example of three data series each with 10 measurements representing frequency of the occurrence of something I'm measuring might be something like this:

a: [10% 11% 10%  9%  9% 11% 10% 10% 12%  8%]
b: [10% 10% 10%  8% 10% 10%  9%  9% 12%  8%]
c: [ 3%  2% 60%  2%  3%  7%  6%  5%  5%  7%]   <-- non-uniform
d: [98% 97% 99% 98% 98% 96% 99% 96% 99% 98%]

I'd like to be able to distinguish distributions like c from those like a and b, and measure c's deviation from a uniform distribution. Equivalently, if there's a metric for how uniform a distribution is (std. deviation close to zero?), I can perhaps use that to distinguish ones with high variance. However, my data may just have one or two outliers, like the c example above, and am not sure if that will be easily detectable that way.

I can hack something to do this in software, but am looking for statistical methods/approaches to justify this formally. I took a class years ago, but stats is not my area. This seems like something that should have a well-known approach. Sorry if any of this is completely bone-headed. Thanks in advance!

Answer 1

If you have not only the frequencies but the actual counts, you can use a $\chi^2$ goodness-of-fit test for each data series. In particular, you wish to use the test for a discrete uniform distribution. This gives you a good test, which allows you to find out which data series are likely not to have been generated by a uniform distribution, but does not provide a measure of uniformity.

There are other possible approaches, such as computing the entropy of each series - the uniform distribution maximizes the entropy, so if the entropy is suspiciously low you would conclude that you probably don't have a uniform distribution. That works as a measure of uniformity in some sense.

Another suggestion would be to use a measure like the Kullback-Leibler divergence, which measures the similarity of two distributions.

Answer 2

In addition to @MansT 's good ideas, you could come up with other measures, but it depends on what you mean by "non-uniformity". To keep it simple, let's look at 4 levels. Perfect uniformity is easy to define:

25 25 25 25

but which of the following is more non-uniform?

20 20 30 30 or 20 20 25 35

or are they equally non-uniform?

if you think they are equally non-uniform, you could use a measure based on the sum of the absolute values of the deviations from normal, scaled by the maximum possible. Then the first is 5 + 5 + 5 + 5 = 20 and the second is 5 + 5 + 0 + 10 = 20. But if you think the second is more nonuniform, you could use something based on the squared deviations in which case the first gets 25 + 25 + 25 + 25 = 100 and the second gets 25 + 25 + 0 + 100 = 150.

Answer 3

Here is a simple heuristic: if you assume elements in any vector sum to $1$ (or simply normalize each element with the sum to achieve this), then uniformity can be represented by L2 norm, which ranges from $\frac{1}{\sqrt d}$ to $1$, with $d$ being the dimension of vectors.

The lower bound $\frac{1}{\sqrt d}$ corresponds to uniformity and upper bound to the $1$-hot vector.

To scale this to a score between $0$ and $1$, you can use $\frac{n*\sqrt d - 1}{\sqrt d - 1}$, where $n$ is the L2 norm.

An example modified from yours with elements summing to $1$ and all vectors with the same dimension for simplicity:

0.10    0.11    0.10    0.09    0.09    0.11    0.10    0.10    0.12    0.08
0.10    0.10    0.10    0.08    0.12    0.12    0.09    0.09    0.12    0.08
0.03    0.02    0.61    0.02    0.03    0.07    0.06    0.05    0.06    0.05

The following will yield $0.0028$, $0.0051$, and $0.4529$ for the rows:

d=size(m,2); 
for i=1:size(m); 
    disp( (norm(m(i,:))*sqrt(d)-1) / (sqrt(d)-1) ); 
end

Answer 4

Stumbled upon this recently, and to add to the answer from @user495285, as far as I understand it:

When the values are normalized and sum to one, then the uniform distribution is the unit sphere in $\mathbb{R}^n$, and what is being calculated by using an $L_p$ norm is the deviation from the unit sphere using a distance measure of a given $p$, i.e. deviation from the uniform distribution in $\mathbb{R}^n$ with geometric distance measure $p$.

The $L_2$ norm places higher weight on large deviations from the unit sphere in any given dimension, whereas smaller values of $p$ place less weight on large deviations.

When the underlying distribution is the unit sphere, the numerator equals zero in the following equation: $$\frac{n\sqrt{d} - 1}{\sqrt{d} - 1}$$ where $n$ is the $L_2$ norm and $d$ is the vector length.

I believe that the usefulness of geometric measures applies when each position (dimension) of the space described is assumed to be measured on equivalent scales, e.g. all counts of potentially equal distribution. The same assumptions underlying change of bases like PCA/SVD probably are similar here. But then again I'm no mathematician, so I'll leave that open to the more informed.