Desired effect size vs. expected effect size

Question

When estimating the effect size for determining optimal sample size for a statistical test, the textbook approach seems to do this from pilot studies or the like. Is it also legal to define effect size as the smallest relevant effect, independent of real, systematic effects which are just too weak to be interesting?

For example, if I want to test an improvement to some existing product which will increase its production cost by some amount. I know that customers will pay this premium if the product is at least x% "better" (by some metric) than the existing solution. Can I use effect size and resulting sample size as tools to suppress significant results for smaller differences? If not, what would be the problem and how would a correct solution look like?

Answer 1

There are many ways to perform power calculations to determine the ideal sample size. One such way, as you mention, is to pick a threshold value of "Someone will care". This may be governed by prior knowledge, the belief that your field doesn't particular get interested until an effect is of a particular size, etc.

My suggestion is instead to look at a range of possible effect sizes. It is trivially easy, if you are performing one power calculation, to perform several to evaluate the sensitivity of your threshold. For example, will you really be alright with your study being underpowered if you set your threshold at 5% and it turns out to be 4.9%? Or 3%. Or a real effect at all?

The figure below for example is asking (for a fixed sample size in this case), what the power of the study is under a range of possible Exposed:Unexposed ratios and effect sizes. It would be just as easy to compose a similar plot varying study population to more fully understand your study's power.

You also seem to be asking if you can use a small sample size to "suppress" particular results. You can design a study and acknowledge that sample size constraints will result in it being underpowered for particular effect sizes, but I wouldn't deliberately do so. A null or small effect finding is still a finding, and deliberately underpowering a study seems...flawed. Also, the optimal sample size obtained from power analysis is often somewhat optimistic - it hardly ever includes missing data, an interesting sub-group analysis, or other problems that will necessitate a bigger sample. Drawing a hard line is a bad idea.

Answer 2

@EpiGrad makes some nice points here. I thought I would throw in one extra bit of information. He mentions that power analyses are often optimistic, and you mention that it is often suggested to estimate the effect size you want to capture from a pilot analysis. I just want to make it explicit, that this procedure is known to be biased such that the estimated power will be higher than the true power. There is considerable uncertainty in the estimated effect size based on a pilot study, or even a full study. If the true effect size is larger than the estimate, you will have more power, and if it is smaller, you will have less power, than you believe. However, the rate of change in power is not the same in both directions. Specifically, power diminishes faster as the effect size shrinks than it increases as the effect size expands. In this way, even though the distribution of effect sizes is symmetrical, the distribution of power is not. To compute a more robust estimate of power, compute the full distribution for the effect size and integrate over it. (A quick and dirty approach is to compute a 95% CI for the ES, and get a weighted average of the power at your estimate and at the CI endpoints.)