10

It's usually straightforward to do a Power Analysis to compute the minimum sample size, especially in R which is my preferred statistical computing environment.

However, I am being asked to conduct a Power Analysis that's a little different than anything I've done or which I can find reference to online. I'm wondering if what I'm being asked for is even possible/valid.

The project basically has two unequal groups of states and the hypothesis is that these two groups are significantly different in terms of an outcome variable (which is the duration of phone calls to customers). The "control" group consists of 40 states and produced about 2,500 observations. The "test" group has about 10 states and 500 observations.

Initially, I found group means + pooled standard deviation, which I used to calculate an Effect Size. Then I used a package called pwr in R and found that I needed a minimum sample size of about 135 observations per group, given .05 significance and .8 power.

However, they rejected my answer because they want one group to be much bigger than the other like it is now, and they are expecting either two different minimum numbers of observations per group or a minimum % of the population in terms of numbers of states or observations that have to go into their "test" group.

I see Power Analyses for two sample t-tests (i.e. the R function pwr.t2n.test), but I'd have to specify at least one of the sample sizes whereas they want me to tell them the minimal sample size for both groups (either as numbers or percentages) and this function doesn't reflect the differences in standard deviations for the two groups.

Is this possible or do I just tell them that's not how it works (i.e. the best I can do is tell them that given one of the sample sizes and a pooled standard deviation the second group has to be at least a certain size)?

Hack-R
  • 848
  • 9
  • 24

2 Answers2

7

You can do sample size calculations for unequal sample sizes.

For example, you can decide the n's are in some ratio (such as in proportion to the populations perhaps).

It's then possible to do power calculations (at the least you can simulate to obtain the power under any particular set of circumstances, whether or not you are able to do the algebra).

The problem is that it's relatively inefficient at finding differences compared to the same total number of observations at equal sample sizes.

Imagine you had a total sample of $n=n_1 + n_2$, with equal variance in the population and close to equal sample variance, and that your choice was between a 50-50 split and a 90-10 split ($n_1 = 0.5n$ vs $n_1=0.9n$).

The two-sample t-statistic is:

$t = \frac{\bar {X}_1 - \bar{X}_2}{s_{\text{pooled}} \cdot \sqrt{\frac{1}{n_1}+\frac{1}{n_2}}}$

The impact of the sample size is in the term $1/{\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}}$.

If you have the 50-50 split it's like having a 40% smaller standard deviation; at a given $n_1+n_2$ you can pick up a substantially smaller effect with the even split.

If the combined sample size is not an effective constraint, this calculation may pointless however. It matters in cases where every observation carries the same marginal cost, which is not always relevant.

Glen_b
  • 257,508
  • 32
  • 553
  • 939
  • Thanks! So, to actually carry this out is my best bet to simply use something like pwr.t2n.test() in R for basically every possible combination given a combined sample size, or is there a better way you'd recommend to compute this? If you use another language/package then I can figure out how to translate your answer into R. In either case, thanks for your solution. – Hack-R Jul 15 '14 at 20:29
  • I don't understand what you're getting at here. As far as I can tell you don't have the ability to split in any ratio you like; my example was to illustrate the effect of uneven splits. If you do have a fixed total sample size, then split it as near to 50-50 as would be accepted. If you don't have a fixed total sample size, then you optimize based on however your costs and constraints actually do go. – Glen_b Jul 15 '14 at 20:33
  • The extent of my information is that they currently have those two really unequal groups which total to 3,000 observations across 50 US states. They ask for the minimum number of observations per group and/or the % which should go into each group, based on the means and standard deviations I get from the data. I have no idea if they could split the data 50-50 or any way they like, but I was going to show them the minimum n in group 2 for every value of group 1's n and also show them the cost of splits other than 50-50, probably in terms of power. I thought that was roughly what you suggested? – Hack-R Jul 15 '14 at 20:43
  • I see. You could certainly calculate something like that, but you wouldn't necessarily need to do it at every $n$; these things progress quite smoothly. – Glen_b Jul 15 '14 at 20:46
  • @MichaelMayer Yes, I made no attempt to deal with that issue at all, but the sample size issue will be similar across analyses. – Glen_b Jul 15 '14 at 21:26
  • 1
    @Glen: That is why I tried to quickly remove my comment about the (un)usefulness of a t test in duration analysis. But you were even faster. Another concern is the grouped structure of the data. But your answer exactly fits to the question. – Michael M Jul 15 '14 at 21:32
  • Don't use $s_{\textrm{Pooled}}$. If the sample sizes and variances are substantially unequal your confidence intervals will be either optimistic or conservative. Use $$s^2_{\bar{x}_1-\bar{x}_2} = \frac{s^2_1}{n_1} + \frac{s^2_2}{n_2}$$. Because you are using sample estimates, figure Satterthwaite's approximation to the DF and use that for your non-central $T$. – Dennis Jul 15 '14 at 23:25
  • @Dennis My discussion specifically assumed equality to make the *illustration of the point simple*. The same underlying issue applies to other analyses, but saves having to consider a variety of cases that don't serve to make the basic issue any clearer. I was not advocating any particular analysis - as Michael Mayer already suggests, t-tests aren't likely to be suitable in the first place, so arguing for a better t-test neither helps the basic explanation of the issue with sample size, nor arrives at a suitable analysis for the problem. To what end, then? – Glen_b Jul 15 '14 at 23:43
  • @Glen_b: there is such a thing as oversimplification. Users don't need any more encouragement to use pooled $T$ than is already out there on their bookshelves. In the case of equal or nearly equal sample sizes, I don't worry too much. In the case of very disparate sample sizes, I do worry. The situation here is to determine sample sizes intended to be disparate. Assuming $\sigma$ known also simplifies the situation tremendously. That would have made the same point, but be nearly impossible to mistake as a suggestion. – Dennis Jul 16 '14 at 01:05
2

First off, why are you assuming equal variances in the two groups? Please don't say, "Because it's convenient." I seriously doubt that the group variances are equal, although in the case of equal sample sizes that isn't crucial. Your degrees of freedom will be off, but you know you have at least 130, so who cares? There are much bigger questions to address.

If you are going to permit (or require) unequal group sample sizes, the problem will not have a unique solution. There are two unknowns ($n_1$ and $n_2$ and only one constraint (the power must be at least $\phi$.) I don't think the problem can be solved without an additional constraint. There are two obvious possibilities. The first is to fix one of the sample sizes (e.g., the sponsors want at least 300 observations from Group I). The other is to fix the ratio (e.g., because Group I is ten times the count of Group II, we want $n_1 = 10\, n_2$). Now proceed with your power analysis.

Dennis
  • 1,710
  • 9
  • 15
  • How did I assume that there were equal variances? I know there's not... that's why I was asking about the possibility of using the different standard deviations for each group to do a calculation, though the normal way of doing a Power Analysis is to use pooled standard deviations when you have more than 1 group. – Hack-R Jul 16 '14 at 13:08
  • @NerdLife: Pooled variance, which you said you used, amounts to assuming equal variances. The "normal" way of doing it is to assume that $n_1 = n_2$, and in that case pooling doesn't matter to the value of the test statistic. The OSL is affected because the degrees of freedom shift between$\n_1-1$ and $2\,(n_1 - 1)$. But those shifts are pretty minor above 30 or so df. – Dennis Jul 16 '14 at 21:19