2

If we have a balanced factorial designed experiment where each variable is taken in 2 levels (+1,-1) and we don't have estimates of each proportion for each factor level combination like we did in this question: Simulation of logistic regression power analysis - designed experiments, what is the best approach for determining sample size? Can we consider the baseline proportion (say 0.0005) and how large a deviation we care to detect (say 0.00005) and then simply run a two sample difference in proportions power analysis? That is: 

power.prop.test(n=NULL, p1=0.0005, p2=0.00055, power=0.8)

which in this case suggests the total sample size required is 6589596 (3294798 *2 from the output of the R function), and divide this sample size between each design point? So if there are 4 factors at 2 levels each then each "cell" gets (6589596 / 16)?

The thinking: Given a factorial design, each effect (main as well as interactions) will use 1/2 of the total sample for (+ levels) and the other half for (- levels) if we say we only care about effects that are at least a certain size, this approach should work regardless of the number of 2 level factors and works for both main effects and interactions.

Is this correct?

Community
  • 1
B_Miner
  • 7,560
  • 20
  • 81
  • 144
  • Possible duplicates: https://stats.stackexchange.com/questions/162294/multiple-logistic-regression-power-analysis/396681#396681 and https://stats.stackexchange.com/questions/35940/simulation-of-logistic-regression-power-analysis-designed-experiments and https://stats.stackexchange.com/questions/26016/sample-size-for-logistic-regression – kjetil b halvorsen Mar 12 '19 at 11:30

2 Answers2

5

As I discuss in my answer to your linked question, there are different kinds of power when there are multiple hypotheses you want to test. For example, you can talk about the all effects power, the power to detect a specific effect, or the any effect power (these are in weakly descending order). If you only care about the one effect, and the other effects are nuisances, you can do what you suggest. (Technically, you should add a couple of additional data to account for the degrees of freedom that will be lost accounting for the nuisance parameters, but that seems inconsequential in your case with so much data anyway.)

On the other hand, if you care about all of these effects, and they are orthogonal (as suggested by "a balanced factorial designed experiment"), then you could do what you suggest for each effect. The all effects power would be the product of the powers for the three specified effects. For instance, let's say that at a given N, the prespecified single effect powers are .82, .80, and .67. Then the power to detect all three would be .82 * .80 * .67 = 0.44.

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
  • yes, assuming an orthogonal factorial designed experiment - is this product of the powers to adjust for multiple testing only? – B_Miner Dec 26 '16 at 21:27
  • It depends on what you mean, @B_Miner. This is about multiple testing, but it doesn't account for any multiple testing correction (eg, Bonferroni); you'd have to account for that in addition. Rather, you are tossing several coins, & you want them all to come up heads--it's harder than when you only toss 1 coin. If you care about all coming up heads, you need (a lot) more N. See the simulated power estimates at the bottom of my answer to your linked Q. – gung - Reinstate Monica Dec 26 '16 at 21:32
  • I was wondering about the multiplication of the three power estimates and if that was being done to show that there is multiple testing happening, sounds like that is what you meant. So, if we say we only care about effect size (difference in proportions) of 'x' (because anything smaller than x is not of clinical or business significance) then, this result says that there are no limits to the number of factors (all at two levels) assuming a balanced design, as long as we can logistically construct all the combinations - as the number of factors increase, the number of needed sample does not! – B_Miner Dec 26 '16 at 21:40
  • The above being said, keeping in mind the multiple testing issue of course! – B_Miner Dec 26 '16 at 21:42
  • @B_Miner, if you only care if 1 prespecified effect is significant (ie, it's the only 1 you're testing in a meaningful sense), & the rest are nuisance parameters, then they largely don't matter to your power calculation. Your model will estimate additional parameters, even if you think they're nuisances, so you should add 1 extra datum per degree of freedom consumed, but +1 is inconsequential when N=6589596. The other thing to consider is the possibility of complete separation & what you'd do about it, if you have lots of factors, but that's not quite the same consideration as power. – gung - Reinstate Monica Dec 26 '16 at 21:46
  • Sorry, no I do care about all the factors - so multiple testing protection is needed (adjust alpha when looking at the power of the worst case difference between two proportions). – B_Miner Dec 26 '16 at 21:49
  • Well, maybe you should, @B_Miner, but that's a different issue. What I'm referring to here is that if you want all heads, & you flip 1 coin, you have a 50-50 shot, but if you flip 2 coins, you've got a 25% chance of all heads. See my answer to your linked Q, & the simulated power estimates at the bottom, I discuss these ideas thoroughly there. – gung - Reinstate Monica Dec 26 '16 at 22:05
  • Are you saying that multiple testing is a different issue? I was thinking that to protect the "all effect" testing as you put it, one should adjust the beta in the power analysis to take into account the multiple effects I cared about (I said alpha above but should have said beta). Is this correct? – B_Miner Dec 26 '16 at 23:45
  • I don't understand your question, @B_Miner. When there is multiple testing, what most people think of (really the only thing most people ever think of) is type I 'familywise' error inflation. People will use, eg, the Bonferroni correction to deal w/ that. What I'm talking about is type *II* error inflation, which will occur b/c of the mathematics of this whether something like the Bonferroni correction is used or not. – gung - Reinstate Monica Dec 26 '16 at 23:54
  • Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/50751/discussion-between-b-miner-and-gung). – B_Miner Dec 27 '16 at 00:24
1

I agree with Gung's answer, but here are a couple of additional thoughts.

Like in the linked question, I would suggest doing some simulations. Simulations give defensible answers, but they also help with understanding.

To do the simulation correctly you need to decide how you plan to analyze the data and what you will consider a "success". As Gung points out, this may be all effects significant, one specific effect significant, or any effect significant (or others), but this needs to be built into the simulation function (you can simulate under each of the conditions to see how they compare).

If you don't know the effect size then you can either rely on expert knowledge to determine either what is likely, or better, what the Minimally Interesting Difference would be (the smallest difference that would be practically significant). You can also calculate power or sample size for various effect sizes and plot them to get a feel for the relationship and show those curves to an expert to help decide on a final sample size. Parallel processing will be a great advantage if doing multiple effect sizes and simulating at each step.

Greg Snow
  • 46,563
  • 2
  • 90
  • 159
  • Thanks Greg! When we can hypothesize "response rates" to each design point, like my linked question, the simulation is more straight forward. It seems like in a lot of cases we really cant (especially for interactions). So it comes down to how to proceed if all you know is the base response and what kind of an increase is 'clinically interesting". Getting that total N and then dividing across all the design points is what I found from research is possible (regardless of the number of factors - which was surprising to me!). – B_Miner Dec 27 '16 at 21:49
  • Gung agreed that works but made the great point I did not consider was that since we usually care about 'all' effects there is a price to pay for increasing the number of factors in the factorial - namely the considering of not having a protected power for all effects. That made me think there is benefit from fractional factorials (which I at first thought were worthless if I could add design points without any down side) – B_Miner Dec 27 '16 at 21:51