Uniform distribution testing

Question

I'm trying to determine the best way to check if a sample of angles is uniformally distributed. As such I've built two toy functions to see what type of sampling error to expect from actually uniform distributions using scipy.stats and numpy.random.

import scipy.stats as stat
import numpy as np

def chiTest(n = 400, d = 10):
    theta = np.random.rand(n) * np.pi - np.pi/2 
    #theta should be uniform on [-pi/2, pi/2]
    bins = np.linspace(-np.pi/2, np.pi/2, n//d + 1)
    d, _ = np.histogram(theta, bins)
    return stat.chisquare(d)[-1]

Chi-squared gives me a p-value that is all over the place:

chiTest()
Out[297]: 0.41045635844052125
Out[298]: 0.78687422600627477
Out[299]: 0.016802707521273268
Out[300]: 0.66269332328844976

While Kolmogorov-Smirnov

def ksTest(n = 400):
    theta = np.random.rand(n) * np.pi - np.pi/2
    uni = np.linspace(-np.pi/2, np.pi/2, n)
    return stat.ks_2samp(theta, uni)[-1]

Is a bit better, but still gives false negatives

ksTest()
Out[350]: 0.31152830907597184
Out[351]: 0.93696634517876642
Out[352]: 0.56898463914998687
Out[353]: 0.74776878262002455

And gets even worse when my "control" distribution is actually random:

def ksTest2(n = 400):
    theta = np.random.rand(n) * np.pi - np.pi/2
    uni = np.random.rand(n) * np.pi - np.pi/2
    return stat.ks_2samp(theta, uni)[-1]

ksTest2()
Out[373]: 0.061328687673192099
Out[374]: 0.68866738727690713
Out[375]: 0.93696634517876642
Out[376]: 0.27118717204504289

The strange thing is that increasing n only seems to help ksTest(), but even big (1000000) sample sizes still can often throw p-values < 0.1. Am I misusing, incorrectly implementing, or just plain misunderstanding these methods? Or is this the best accuracy I can hope for considering the possibility of random clustering?

Regardless of sample size, a test of the null hypothesis is *supposed* to give p-values "all over the place"! They themselves should be uniformly distributed. A chi-squared test is an excellent choice for testing uniformity of angles: just make sure you don't let the software choose the bins. You should select them based on no knowledge of the data (apart from their count, to make sure the bin counts are large enough to make the chi-squared distribution applicable). — whuber, Aug 17 '17 at 13:40
I'm sorry, I just can't wrap my head around how chi-squared can be an excellent test of uniformity, when it is also *supposed to* yield random p-values when testing for uniformity. — Daniel F, Aug 17 '17 at 14:01
That is how *all* hypothesis tests work. Under the null hypothesis, $p$ is uniformly distributed. (I fudge a little, because there are exceptions for discrete distributions--but they don't change the basic idea.) Under alternative hypotheses, if your procedure is any good then $p$ will be concentrated towards smaller values. If that's not completely clear, then please study hypothesis testing before proceeding. Here's a good set of links: https://stats.stackexchange.com/search?q=hypothesis+test+p+value+uniform. — whuber, Aug 17 '17 at 14:14
But . . then how do you test if $p$ is uniformly distributed? It seems like a recursive problem. So a dataset $x$ is uniform if and only if $f(x)$ is uniform, iff $f(f(x))$ is uniform, iff $f(f(f(x)))$ is uniform . . . — Daniel F, Aug 17 '17 at 14:50
There's no recursion. There's just one dataset. A test is applied exactly once and its p-value is compared to the threshold of significance you choose. Done. — whuber, Aug 17 '17 at 15:12
But you just said the p-value will be uniformally distributed over $[0,1]$, so what is a suitable threshhold? — Daniel F, Aug 17 '17 at 17:50
You seem to be confusing uniformity of the data at hand and uniformity of the p-value under the null hypothesis. p-values are uniform under the null essentially by construction. This is a mathematical fact, not something to be tested for. — klumbard, Aug 17 '17 at 17:56
So basically I can't test for uniformity, I can only test for non-uniformity and suck up a false positive rate equal to whatever my p-value threshold is? Basically my question was always "Am I implementing this incorrectly, or expecting too much from it?" - and I guess I'm expecting too much. — Daniel F, Aug 18 '17 at 06:28
Well, pretty much. But that's not a feature of your specific question, that is how any hypothesis test works. We don't "accept" the null hypothesis if p > threshold, we can only say our observed data are consistent with the null hypothesis (i.e. "fail to reject"). I haven't looked at what you're doing specifically mostly because I'm too lazy to parse the Python code. Hence why I'm in the comments and not answering the question. If you fail to reject the null but don't like that we're not "accepting" it, then look into effect sizes for goodness of fit tests to quantify things more explicitly. — klumbard, Aug 18 '17 at 13:19
Basically, see : https://stats.stackexchange.com/questions/114027/distribution-hypothesis-testing-what-is-the-point-of-doing-it-if-you-cant-ac/114029#114029 — klumbard, Aug 18 '17 at 13:41

score 2 · Accepted Answer · answered Aug 17 '17 at 11:24

2

Use a Kuiper test. It's location shift invariance makes it particularly suited to testing circular distributions.

answered Aug 17 '17 at 11:24

David Wright

2,181
12
12

score 0 · Answer 2 · answered Aug 17 '17 at 10:34

My current workaround is to fit the data to a von Mises distribution, and if the k-value is suitably small, assume that the distribution is uniform.

def vmTest(n=400):
    theta = np.random.rand(n) * np.pi - np.pi/2
    k, t, _ = stat.vonmises.fit(theta * 2, fscale = 1)
    return k

This rules out a single bias direction, but not, for example, two orthogonal biases.

Uniform distribution testing

2 Answers2