2

Suppose I have two samples with the following statistics and I want to determine if there is a difference in their means.

First:

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    0.0    39.0    39.0    38.6    40.0    48.0

Others:

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   4.00   39.00   39.00   38.51   39.00   50.00

Performing a t.test in R I get:

t.test(first, other)

        Welch Two Sample t-test

data:  first and other
t = 1.3771, df = 8981.579, p-value = 0.1685
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.03304794  0.18912248
sample estimates:
mean of x mean of y 38.60095  38.52291

If our $\alpha$ was 0.05 we would not reject the null hypothesis.

But suppose we assume that the means are not the same and the actual difference in the means is 0.078. How would I go about computing the power of this test given an $\alpha$ of 0.05?

I believe I can approximate power by sampling from the distributions and comparing the means and counting the number of times the null hypothesis would be rejected with a given $\alpha$ but I want to confirm that my simulation is correct.

Following @whuber's suggestion from this post I tried computing the power using the following:

first.mean = 38.60095
first.sd = 2.791901
first.count = 4413

other.mean = 38.51326
other.sd = 2.634525
other.count = 4735

z = replicate(10^3, {
  first = rnorm(first.count, mean=first.mean, sd=first.sd)  
  other = rnorm(other.count, mean=other.mean, sd=other.sd)  
  t.test(first, other)$p.value
})
length(z[z < 0.05])/length(z) # mean(z < 0.05)

I get 0.30 (thanks for catching the typo @John). Is there a way to confirm this analytically?

This is from Allen Downey's ThinkStats book chapter 7 exercise 7. I took a stab at simulating the power and now want to confirm my result analytically if possible.

Robert Long
  • 53,316
  • 10
  • 84
  • 148
Damon Snyder
  • 173
  • 1
  • 5
  • You haven't actually described a hypothesis test yet. What "test" are you referring to? Incidentally, just a few hours ago I posted a power calculation for a t-test of means at http://stats.stackexchange.com/questions/69898/t-test-on-highly-skewed-data/69967#69967. The last half dozen lines of code (at the end) do the calculation. An analytic computation of the power will require making some distributional assumptions in addition to stipulating a particular test. – whuber Sep 13 '13 at 20:28
  • 1
    Hi @whuber thanks for commenting. I'm new to statistics and I don't yet have a firm grasp on the language of stats. I updated the question to provide the t-test and some summary data as well as an attempt at the simulation that referenced. I realize that this test may also suffer from the same shortcomings (not enough observations) as the question you referenced. I hope that working through it will improve my intuition. Thanks! – Damon Snyder Sep 16 '13 at 16:17
  • Your power is much higher than 0.07. You've got an error in sampling `other` where the sd should be `other.sd`. I get about 0.35 then. (And this is a great way to calculate power. There are situations where there is no analytic method where this will work just fine.) – John Sep 16 '13 at 17:05
  • Doh! Thank you @John. I corrected it above and got about the same. – Damon Snyder Sep 16 '13 at 19:08
  • Another tip, run simulations a few times or for at least more than 1000 runs... 10000 in this case doesn't take long. – John Sep 16 '13 at 19:55
  • Post-hoc "power" calculation doesn't really tell you anything meaningful – Glen_b Nov 09 '18 at 10:31

0 Answers0