2

Suppose one would like to test some new hypothesis, for which there are no previous data available. To estimate the needed sample size, one should do a power analysis. Since there are no previous data available, the power analysis can only be carried out by conducting a pilot study first. How does one choose the sample size for a pilot study?

The field is neuroscience, where sample sizes are generally small due to financial and other constraints (cost per measured subject is easily in the thousands).

If I'd like to compare two groups, would a pilot study with 4+4 participants be enough to run a power analysis? How does the sample size of the pilot study affect the reliability of the power analysis?

Are the there any techniques (e.g. bootstrap?) that I could use to improve the power analysis?

ada
  • 21
  • 2

2 Answers2

3

A pilot study is a good idea if an experimenter does not have recent historical data that can be used to estimate sigma. The effect that you want to detect should be the smallest effect that has practical significance. You don't need a pilot study to determine that. Pilot studies are generally small. You want to have enough observations so that you can estimate sigma with reasonable precision, as a poor estimate will undermine the hypothesis test.

user44867
  • 29
  • 1
0

If you want to compare two groups, you can do a power analysis without a pilot study! See for example: http://www.ats.ucla.edu/stat/stata/dae/t_test_power2.htm

Conversely, if your analysis should go beyond a simple treatment-control comparison, you could simulate data and analysis: Power analysis for ordinal logistic regression

Community
  • 1
stochazesthai
  • 4,616
  • 2
  • 18
  • 26
  • 1
    Power analysis needs an estimate of effect size. Without any previous data, one does not know it. – mmh Jun 20 '15 at 13:07
  • Yes, but I mean you can do an hypothesis on the expected effect size! – stochazesthai Jun 20 '15 at 13:15
  • 1
    Moreover, if the sample size is 4 per group the effect size needed to achieve a 0.8 power should be huge... – stochazesthai Jun 20 '15 at 13:16
  • 1
    If you are studying something new, how could you have an expected effect size? It would be just pure guessing without a pilot study. Though, I don't know how to answer the question. – mmh Jun 20 '15 at 13:16
  • 1
    Well, you are designing an experiment so that IF you hypothesize a certain effect size, you need a sample size pf N to achieve a power of 0.8 and a significance level of 0.95. I mean when you are designing an experiment for a new drug that should reduce the mortality you don't have previous studies but you guess an effect size in order to design the experiment. – stochazesthai Jun 20 '15 at 13:22
  • Thanks for explaining. I think I got your point now (reading the answer by @stockhazesthai helped as well). – mmh Jun 21 '15 at 07:46
  • In the page linked from the first paragraph of this answer, there is prior information not only on the effect size but also on the standard deviations of the values, and there is an assumption of normally distributed values. Simulations also require similar information and assumptions, although the yes/no dichotomy of responses analyzed by logistic regression may simplify this process a bit. – EdM Jun 21 '15 at 11:34