3

I generated p-values from 10K trials with null hypothesis set to true. My goal was to see what the distribution of random p-values is like. The sample size of each test was 2000 with a 5% success rate. p-values calculated using "N-1 two proportion test" based on normal distribution (Sauro 2012). Here's the formula:

N-1 test

Are there ranges of p-values that are unlikely, as this chart shows? Or is there some flaw in my method?

I do remember reading something about p-values below 0.05 being more likely than around 0.05, but I don't see that here. Instead I see this banding closer to the high p-values.

Random p-values

I also plotted the density of p-values. This has minor waviness at the high end of the chart, which I guess corresponds to the big bands. The density chart usually shows a slight peak at 0.1.

Density plot of random p-values

Can anyone confirm any of this and/or point me to some info on expected distribution of p-values?

Vlad
  • 391
  • 2
  • 7
  • 3
    What is a "null test" and how exactly did you compute the p-values? – whuber Nov 01 '15 at 23:36
  • More details of what you did are needed. Were these chi-square tests by any chance? – Glen_b Nov 02 '15 at 00:46
  • 1
    By "null test" I meant "experiment where null hypothesis is true". I calculated p-values using the "two proportion test" which is mathematically equivalent to chi^2 but uses normal distribution. – Vlad Nov 02 '15 at 01:00
  • I edited the first paragraph with link to formula used. – Vlad Nov 02 '15 at 01:06
  • Your link gives "You have reached your viewing limit for this book"... which is no help at all. Please edit to give the essential information in your question (at the least, copy your comment up). – Glen_b Nov 02 '15 at 02:29
  • 1
    [This post](http://stats.stackexchange.com/questions/153249/distribution-of-p-values-binomial-test/153267#153267) discusses where the bands originate. If you need something not covered there, you may want to refine your question. – Glen_b Nov 02 '15 at 02:33

0 Answers0