2

I'm trying to understand what's happening in the article Using simulation to estimate the power of a statistical test

I'm familiar with bootstrapping which makes sense to me. But I don't understand how bootstrapping can be used to determine power.

From the article

  1. Simulate data from the model for each group's population. These are the samples. The populations are chosen so that the true difference between the population means is δ > 0. (The null hypothesis is false.)
  2. Run the TTEST procedure on the samples. For each sample, record whether the t test rejects the null hypothesis.
  3. Count how many times the t test rejects the null hypothesis. This proportion is an estimate for the power of the test.

Is counting how many times the test rejects the null hypothesis all you need to determine power? That doesn't make sense to me.

Unclear
  • 21
  • 1
  • 2
  • 2
    Well, what you really care about is the fraction of times the test rejects the null. This is an estimate of the power of the test at whatever difference $\delta$ was specified in the simulation. The actual power of the test at $\delta$ is the probability that the test will reject the (false) null hypothesis when the true parameter value is $\delta$, and you should be able to see why the simulation gives you an estimate of that probability, just as the fraction of heads that comes up in, say, 1000 coin flips gives you an estimate of the probability that the coin will come up heads. – jbowman Oct 01 '18 at 16:38
  • Yes, "power" is the "chance of rejecting the null hypothesis if the null hypothesis is false." By simulating the data with a known difference (aka, based on the alternate hypothesis being true), followed by a test, we can see if the null is indeed rejected. The proportion of simulated tests showing a rejection is then "the chance of rejecting the null if it is false", aka the power. – Penguin_Knight Oct 01 '18 at 17:48
  • 3
    It might be worthwhile recognizing that although this procedure is simulation, it is not bootstrapping, because the model is *assumed* rather than estimated from data. It looks like a straightforward power study whose methods are determined solely by the definition of power and not by any deeper or more specialized considerations. – whuber Oct 01 '18 at 18:23
  • @whuber is there any chance I could get you to elaborate on your comments in an answer? They've been quite helpful to me. – Unclear Oct 01 '18 at 23:53
  • I think my answer at https://stats.stackexchange.com/a/74136/919 might already provide what you're looking for. Read the final paragraph first to determine whether the rest might be worth studying. – whuber Oct 02 '18 at 13:31

2 Answers2

2

Let's simplify the problem by assuming you are interested in estimating the power of a one-sample t-test for testing a population mean mu via the hypotheses Ho: mu = 0 vs Ha: mu != 0. Assume the population is normal with unknown mean mu and known standard deviation sigma = 1.

To estimate the power of the test via simulation, you would assume that mu = 2, say (or any other relevant value encompassed by the alternative hypothesis and falling in line with the magnitude of mu you would want to be able to detect via the t-test) and generate a large number N of random samples of size n from a normal population with mean mu = 2 and standard deviation sigma = 1. Using the data from each of these random samples, you would perform a one-sample t-test of Ho: mu = 0 vs Ha: mu != 0. The power of the test for rejecting Ho: mu = 0 in favour of Ha: mu != 0 would be given by the proportion of these tests where H0 was rejected in favour of Ha. The power is indexed by the value of mu used for generating the random samples (in this example, mu = 2).

When you are in a bootstrapping situation, instead of drawing N random samples from the target normal population under the assumption that mu = 2 and sigma = 1, say, you draw a single random sample. Then you treat that sample "as if" it were the entire population and can draw random samples from that assumed entire population using bootstrapping.

In the context of this simplified example,the reason you may want to use bootstrapping for estimating power is because you may actually be in a situation where you won't know whether it's sensible to assume that the underlying population is normal, but you would think that the sample distribution provides a reasonable approximation to the population distribution and since that's all you got, you have to do the best you can with that information - hence, bootstrapping.

Isabella Ghement
  • 18,164
  • 2
  • 22
  • 46
  • 2
    It's hard to carry out a power study using the method you describe, because it does not allow any variation of the effect size. What one might do, subject to specific assumptions about the null hypothesis, is to use residuals from a fit of the data using a "saturated model" to generate data from a range of elements of the null hypothesis (either parametrically or nonparametrically). – whuber Oct 01 '18 at 18:25
  • Does your final paragraph mean using bootstrapping within each replication? If it does, I don't think my answer adds anything here. – Heteroskedastic Jim Oct 02 '18 at 11:57
0

To make the comments into an answer, the three step procedure you laid out is exactly right for calculating statistical power using simulation. Power is the ability to detect an effect assuming it is not zero. Given certain conditions, were you to repeat this process a large number of times, what proportion of the time would you reject the null?

Keeping the same two group design, we can then vary conditions to see their effect on statistical power: the sample sizes for the groups; the magnitude of $\delta$; the variance of the outcome (by group); the $\alpha$-level; the statistical test; distributional assumptions about the outcome (by group); is $\delta$ heterogenous; group classification error; measurement error in the outcome; sampling strategy; ...

All of these conditions have potential impacts on power, so we can vary them and study their impacts. They are assumptions you make a-priori and these assumptions should be informed by some subject matter expertise and a healthy skepticism about effects $-$ we tend to expect effects to larger than they turn out to be.

Finally, this is not bootstrapping. Bootstrapping is used for constructing confidence intervals given model-based estimates. Within the power analysis you described above, you can bootstrap the group comparison to check whether you reject the null; that would be an example of varying the statistical test. Outside of that, I don't see how it helps one do a power analysis.

Heteroskedastic Jim
  • 4,567
  • 1
  • 10
  • 32