1

I'm using a Kolmogorov-Smirnov test to examine how normal (or Gaussian) some data sets I have are. I am having trouble with interpreting the $p$-values (surprise surprise).

My present understanding is that if the returned $p$-value is $p > 0.05$ this suggests that the data is normally distributed, while datasets which produce a Kolmogorov-Smirnov test result $p < 0.05$ suggest that the data is not normally distributed.

Firstly, is my present understanding/interpretation correct?

Secondly I performed some very basic simulations. I simply generate data sets with $1000$ points which are normally distributed and perform a Kolmogorov-Smirnov test on each one. If I look at the distribution of $p$-values extracted from this, I see a square distribution with values ranging between 0 and 1.

I don't understand this -- I guess I don't really understand $p$-values as I expected for data which definitely is normally distributed (I generated it to be such) that there would be a median $p$-value.

Q.P.
  • 248
  • 1
  • 13

1 Answers1

4

You're on the right track. Two facts are key for understanding what's going on.

First, the null hypothesis $(H_\emptyset$) of the one-sample K-S test is that the sample data comes from the "reference" distribution. When it is being used as a normality test, that reference distribution is a Gaussian.

Second, for any test, p-values are uniformly distributed between 0 and 1 under the null hypothesis, assuming the test's assumptions are met. See this question for more details. While you should expect small $p$-values if the alternate hypothesis were true, it is not the case that you should expect only large $p$-values under the null hypothesis.

Combining these two facts accounts for your simulation: you set up conditions where the null hypothesis is definitely true, and the resulting p-values form a "square" histogram where each bin has approximately the same number of values. If you were to draw samples from a different distribution (e.g., an exponential distribution), you would instead find $p$-values clustered near 0, since it is unlikely that those values could have come from a Gaussian (your $H_\emptyset$).

Matt Krause
  • 19,089
  • 3
  • 60
  • 101
  • I think I understand a bit more now. So while I observe a square or uniform distribution, I can say 95% of the data sets satisfy the the K-S tests? And I can safely say there is only a 5% chance a truly normal data set will be rejected? – Q.P. Oct 07 '20 at 14:59
  • @Q.P. What do you mean that 95% of data sets will satisfy the KS test? – Dave Oct 07 '20 at 16:56
  • @Dave I'll illustrate what I mean by adding my simulation and the results. – Q.P. Oct 07 '20 at 17:13