3

I am comparing two approaches A and B using a t-test. I can easily obtain samples using A since the data is generated using a simulator. However, getting a sample using B is time-consuming and costly.

For a scientific paper, I obtained 1000 samples using A and 30 samples using B. Now the reviewers point out that there is a mismatch between the sample sizes. However, as I understand it, in general, "the more data the better." Obtaining 1000 samples using B is not an option for me but I thought it makes sense to get as many samples as possible using A. Would it make sense to run the simulation again for A and just use 30 samples or can I justify the difference between the sample sizes from a statistical point of view?

I looked at How should one interpret the comparison of means from different sample sizes?, where it is stated that different samples sizes don't cause a problem and that the power of the t-test can be increased if the total sample size increases.

Pold
  • 133
  • 5
  • 3
    It's (almost certainly) not a problem. Difference in sample sizes makes the test more sensitive to heterogeneity of variance. I would ask the reviewers why they think it is a problem. – Jeremy Miles Jan 28 '20 at 21:19

1 Answers1

2

There is no problem with unequal sample sizes, and the validity of the t-test you plan to use is not impaired by the unequal sample sizes. The test might be more robust, however, with equal (or similar) sample sizes. With samples from the two groups obtained by different methods, there may be no reason to assume equal variances, so you could use a t-test not assuming equal variances.

In fact, possibly unequal variances should be a more important concern than the different group sizes. All else equal, it would be better to invest in increasing the sample size of the smaller group, but no sense in discarding data already obtained from the larger group.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
  • Let's say you have 30 observations of each group. You have enough resources to collect another 60 observations. Are you saying that the 30/90 distribution of data will have worse robustness than the 30/30, or that 30/90 will have worse robustness than 60/60? – Dave Jan 28 '20 at 21:00
  • I am saying that 30/90 have worse robustness than 60/60. I doubt very much that throwing away data could improve robustness ... – kjetil b halvorsen Jan 28 '20 at 21:09
  • I figured that's what you meant...invest in getting the groups to a more even size, but more data, regardless of group, won't hurt performance. – Dave Jan 28 '20 at 21:19
  • 1
    The combined sample size being equal, balanced groups give more powerful test than unbalanced groups. If you keep one group small and get more and more cases for the bigger group, you increase power slow. It is better to enlarge the smaller group, from all accounts. – ttnphns Jan 28 '20 at 23:06