8

I would like to do power analyses for hypothesis tests of (non-)equality of proportions in which the proportions are very small. I would like to do so without using normal (or Poisson) approximations of the binomial distribution. There are several general types of power questions I'd like to be able to address.

  1. Post-hoc: Given $\Pr_1$ (probability of a success in group 1) and $\Pr_2$ and $N_1$ (sample size group 1) and $N_2$ to calculate the power of the design given $\alpha$.
  2. A priori solve for $N$ given $\alpha$, the ratio $N_1\over{N_2}$, $1 - \beta$ (power), $\alpha$, $\Pr_1$, and an expected $\Pr_2$
  3. A priori solve for $1 - \beta$ given $\alpha, N_1, N_2, \Pr_1$, and $\Pr_2$.

An ideal response would include R code and point out any other givens that I forgot to point out. A simulation approach is not a suitable response due to the small proportions. With your solution, please also mention what kind of statistical test it is applicable to.

russellpierce
  • 17,079
  • 16
  • 67
  • 98
  • One very general way to deal w/ these is via simulation. See, e.g., my answer here: [simulation of logistic regression power analysis designed experiments](http://stats.stackexchange.com/questions/35940//35994#35994). – gung - Reinstate Monica May 26 '13 at 16:42
  • 1
    I didn't think simulations would be a fruitful approach when proportions are *very* small (on the order of 1 in 50,000 or less). – russellpierce May 26 '13 at 17:34
  • I'm also not clear on how hypothetical instances where a marginal proportion is 0 should be treated. – russellpierce May 26 '13 at 17:39
  • @RusselS.Pierce, you're right, simulations become more unwieldy the closer the proportions get to the boundaries. That's because power (viz, whether an iteration is 'significant') is a Bernoulli. – gung - Reinstate Monica May 26 '13 at 17:44
  • This is relevant: https://stats.stackexchange.com/questions/235750/experimental-design-on-testing-proportions/270076#270076 *how small* do you imagine $p$ can be? – kjetil b halvorsen Apr 14 '17 at 12:31
  • @kjetil b Halverson would you please elaborate re: relevance? – russellpierce Apr 14 '17 at 12:35
  • Well, it analyzes two-sample binomial comparisons. You can adapt code from there to make plots for the case where both $p$'s are very small. You say about 1/50000 (or even smaller) then you cannot relly expect to learn much if not $n$ is at least around 150000. – kjetil b halvorsen Apr 14 '17 at 12:39

1 Answers1

1

This is not an answer. It is a community wiki that people may edit as they look for the answer.

G*power 3 can perform (approximations) of these analyses (per this site). The canonical reference for that software provides a reference for performing (at least some) of these types of power analyses as Cohen, 1988 chapter 6 (and 7) as does this example using SAS. The exact equations/procedures may be available from that source. However, the approximations appear to break down at small probabilities.

russellpierce
  • 17,079
  • 16
  • 67
  • 98