0

The question "Test of Equal Proportions with zero successes" showed me that using a test of equal proportions with very low or very high numbers of successes doesn't give an accurate p-value. Below is an example where I'm testing if three independent experiments have the same probability of success ($H_0: p_1 = p_2 = p_3$), for $n_1 = n_2 = n_3 = 10$ and $f_1 = f_2 = 0$ and $f_3 = 1$ (zero successes out of 10 trials in two of the three experiments, one success in the third experiment).

prop.test(c(0,0,1), c(10,10,10))$p.value
## [1] 0.3554102
## Warning message:
## In prop.test(c(0, 0, 1), c(10, 10, 10)) :
##   Chi-squared approximation may be incorrect

The same with $f^\prime = n - f$:

prop.test(c(10,10,9), c(10,10,10))$p.value
## [1] 0.3554102
## Warning message:
## In prop.test(c(10, 10, 9), c(10, 10, 10)) :
##   Chi-squared approximation may be incorrect

How to compute exact p-values in these cases?

krlmlr
  • 749
  • 1
  • 8
  • 35
  • See also [Given the power of computers these days, is there ever a reason to do a chi-squared test rather than Fisher's exact test?](http://stats.stackexchange.com/q/14226/17230), [On Fisher's exact test: What test would have been appropriate if the lady hadn't known the number of milk-first cups?](http://stats.stackexchange.com/q/136584/17230), & [Using brute force to get exact p-value instead of Chi-Squared Test of Independence](http://stats.stackexchange.com/q/116714/17230). But note that the discreteness taken into account in exact tests reduces power, & that conditioning exacerbates ... – Scortchi - Reinstate Monica Aug 20 '15 at 11:33
  • ... this effect; so (considering your example) you can already see what should be plain in any case: that the data furnish negligible evidence against the null hypothesis. – Scortchi - Reinstate Monica Aug 20 '15 at 12:03
  • @Scortchi: Thanks for the reference to Fisher's test. I'm not interested in rejecting the null, I'm collecting vast amounts of p-values to show that they're distributed uniformly to give support **for** the null hypothesis. (Is the [tag:combining-p-values] tag chosen correctly here?) – krlmlr Aug 20 '15 at 14:35
  • @klmlr: Think you need to ask a new question about that - & the combining p-values tag should be appropriate. Do emphasize, though, that the test statistic is discrete. – Scortchi - Reinstate Monica Aug 20 '15 at 15:09
  • @Scortchi: Are you sure Fisher's exact test is applicable here? The column marginals are by no way fixed in my situation. – krlmlr Aug 21 '15 at 09:12
  • That's a vexed question, touched on (barely) in the second link I gave - & note the discussion in the comments. One day I *do* intend to add to that. – Scortchi - Reinstate Monica Aug 21 '15 at 09:17

0 Answers0