1

I am used to performing power analysis with unweighted data. In R, I use the pwr package, which is based on Cohen (1988).

How would you do power analysis on weighted survey data? I don't think you could use the same calculations. The weights would probably somehow have to figure into it? Is there a reference that discusses this?

Jessica
  • 1,019
  • 7
  • 14

2 Answers2

3

The weights do figure in a major way. The design effect (the ratio of variances for the complex design and an SRS that you are used to power-analyzing) due to unequal weights is customarily written as $$ {\rm DEFF}_w = \frac{V_{\rm Design}[y]}{V_{\rm SRS}[y]}=1+{\rm CV}^2_w = \frac{n\sum_i w_i^2}{\bigl(\sum_i w_i)^2} $$ So as a very first approximation, you can compute the expected CV of weights for your study, compute this DEFF, and pro-rate the sample size computed from the simple unweighted analysis by that number. This procedure assumes that the weights are not correlated with your important analysis variables (the outcome itself, the treatment groups, etc.), which may or may not be a reasonable assumption. A more accurate analysis would have to take stratification and other complex sample features explicitly.

As Greg Snow correctly pointed out, a simulation study would indeed be the next logical step in trying to do this with an utmost precision. Doing simulations with unequal probabilities of selection, however, is a fairly complicated business, and if you are new to it, you will likely do it wrong for the first five or so times.

Community
  • 1
StasK
  • 29,235
  • 2
  • 80
  • 165
  • 1
    Sorry to resurrect an old post. What do you mean by "pro-rate" the sample size computed from the simple unweighted power analysis? I'm trying to determine the needed sample size for a chi square using complex stratified data. Does that mean I should do a simple chi-square sample size analysis, then adjust the required sample size with the DEFF? How shall I do that? – JJM Oct 15 '14 at 14:26
  • Your best bet would probably be a simulation. This DEFF is only applicable when the unequal weights is the only complication. With a more complex design, you are looking at generalized design effects and Rao-Scott corrections and what not, and it will likely get too messy to be analytically tractable. – StasK Oct 15 '14 at 16:05
  • 1
    I can't find any resources through Google or CV that explain how to do a power analysis by simulation of complex survey data. Do you know of any online resources, or are textbooks my only hope? – JJM Oct 15 '14 at 18:21
  • If it is not touched upon in the [Practical Tools](http://www.amazon.com/Practical-Designing-Weighting-Statistics-Behavioral/dp/146146448X) book, then you are on your own. – StasK Oct 15 '14 at 20:42
  • @StasK, could you suggest me a citable reference to the formula above (especially to the last formula which takes _weight_ and _sample size (n)_ in the argument)? – Ayalew A. Jan 30 '15 at 08:31
  • 1
    @AyalewA., Korn & Graubard (1999) book talks about this. – StasK Jan 30 '15 at 15:50
1

For power studies beyond the simple 2 sample t-test it is probably best to use simulations to estimate the power.

This Answer gives more detail and an example (though not of weighted regression, but the idea would be similar).

Community
  • 1
Greg Snow
  • 46,563
  • 2
  • 90
  • 159