2

What method is appropriate to use for determining a sample size for one sample t-test? How can I conduct the power analysis in this case? I am looking for the analytical formula that would allow me to compute the required N by hand.

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650

1 Answers1

5

This is a fairly conventional problem. You just need to decide on the effect size you want to be able to differentiate from $0$. That is, how many standard deviations separate the mean of your data from the null (reference) value in your alternative hypothesis? Once you have figured that out, you can use standard methods. The free G*Power software will be very convenient to use.

Below, I determine the required $N$ to differentiate a mean of $145$, with a posited standard deviation of $20$, from a null value of $135$ (i.e., $d=.5$). I am using $\alpha=.05$ with $80\%$ power and intend to run a two-tailed test, all of which is very conventional.

enter image description here

I will need $34$ data points.


Update: I have not found a formula for the power of a t-test that you could use to calculate $N$ by hand. The formula for a one-sample $z$-test (i.e., the standard deviation is known without error) is:
$$ N = \left(\frac{z_\alpha - z_\beta}{\delta}\right)^2 $$ where:

  • $z_\alpha$ is the critical value of $z$ for significance (for a two-tailed test, with $\alpha = .05$, this would be the quantile $\Phi^{-1}(.975) = 1.956$)
  • $z_\beta$ is the normal quantile for $1 - \rm power$
  • $\delta$ is the difference between the mean under the alternative hypothesis and the null value, divided by the standard deviation

This cannot be done with just paper and pencil, but if you had access to a $z$-table, it could be done. Using your values, that rounds up to $N = 31$ (see below). If the suggested number of data were large enough, the normal approximation would be fine; here we are on the border, so we want to take the uncertainty of the standard deviation into account in the resulting test. That is, you will use a $t$-test in the end, not a $z$-test, so you would have a little less power than that. We can try plugging the same values in and using quantiles of the $t$-distribution instead of the $z$-distribution, but we need the degrees of freedom for the $t$-distribution. We can try using $df = 30$, since that falls out of the calculation above. That yields $N = 33, df= 32$, so we didn't use the correct $t$-distribution. If we iterate and try again, the calculation is stable. This is an iterative process, but doable by hand (with a sufficiently detailed $t$-table), and agrees well with the value from G*Power above. Simulation suggests this value rounds to $80\%$ power, but is just slightly below, while $N = 34$ is just slightly above.

za = qnorm(.975);      za  # [1]  1.959964
zb = qnorm(.2);        zb  # [1] -0.8416212
n  = ( (za-zb)/.5 )^2; n   # [1] 30.30848

ta = qt(.975, df=30);  ta  # [1]  2.042272
tb = qt(.2,   df=30);  tb  # [1] -0.8537673
n  = ( (ta-tb)/.5 )^2; n   # [1] 33.54818

ta = qt(.975, df=32);  ta  # [1]  2.036933
tb = qt(.2,   df=32);  tb  # [1] -0.8529985
n  = ( (ta-tb)/.5 )^2; n   # [1] 33.40682

library(binom)
set.seed(4116)
p = vector(length=100000)
for(i in 1:100000){
  d    = rnorm(33, mean=145, sd=20)
  p[i] = t.test(d, mu=135)$p.value
}
mean(p<.05)  # [1] 0.79573
binom.confint(79573, 100000, methods="exact")
# method     x     n    mean     lower     upper
#  exact 79573 1e+05 0.79573 0.7932176 0.7982252

set.seed(4116)
for(i in 1:100000){
  d    = rnorm(34, mean=145, sd=20)
  p[i] = t.test(d, mu=135)$p.value
}
mean(p<.05)  # [1] 0.8093
binom.confint(80930, 100000, methods="exact")
#  method     x     n   mean     lower     upper
#   exact 80930 1e+05 0.8093 0.8068512 0.8117309
Pitouille
  • 1,506
  • 3
  • 5
  • 16
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
  • Glad to see G*Power getting publicity; excellent program and incredibly easy to use. – Chris C Apr 15 '15 at 02:48
  • Thanks. And what if I want to determine the sample size without softvare? –  Apr 15 '15 at 02:49
  • @gil, you want the formula to do it with pen & paper? – gung - Reinstate Monica Apr 15 '15 at 03:00
  • yes... exactly this I would like to know –  Apr 15 '15 at 03:11
  • @gil, I don't know off the top of my head, & it will be a while before I can look it up. I edited the Q to clarify what you want to know. Maybe someone else can provide it. – gung - Reinstate Monica Apr 15 '15 at 03:26
  • I don't think there's an analytical formula that will allow you to calculate the required sample size by hand. – mark999 Apr 15 '15 at 07:49
  • @AleksandrBlekh, the edit is fine but note that '[data](http://dictionary.reference.com/browse/data)' is already plural; 'datum' is the singular. – gung - Reinstate Monica Apr 15 '15 at 14:09
  • @mark999, there is a formula, I just don't know it. G*Power is solving this analytically. – gung - Reinstate Monica Apr 15 '15 at 14:10
  • @gung: It is not quite true in **modern** English. For example, see "Usage discussion of _data_" section in [Merriam-Webster](http://www.merriam-webster.com/dictionary/data) and the corresponding article in [Grammarist](http://grammarist.com/usage/data). Anyway, I just thought that my version sounds a bit better. – Aleksandr Blekh Apr 15 '15 at 16:16
  • @AleksandrBlekh, your version is fine. I am aware that the term is in flux due to the fact that many people use it incorrectly. As statisticians, I believe we should pointed use the term correctly to spit into the wind counteract this. – gung - Reinstate Monica Apr 15 '15 at 16:22
  • @gung: I understand. However, I wouldn't label that use as "incorrect". English, as most languages, is a dynamic phenomenon. Thus, such case can be considered as language development, maturation or act of self-identity. I'm not sure that statistical and, in general, scientific literature should strictly follow their Latin and other roots in regard to terminology. – Aleksandr Blekh Apr 15 '15 at 16:32
  • @mark999: Indeed, there are formulas for that. The approach is [based on non-centrality parameter of non-central t-distribution](http://en.wikipedia.org/wiki/Noncentral_t-distribution#Use_in_power_analysis). I was working on a comprehensive answer with full analytical approach, but put that on hold due to time constraints. I hope to finish that effort eventually, though. – Aleksandr Blekh Apr 15 '15 at 16:34
  • 1
    @AleksandrBlekh, I hope you do. It will be worth having. – gung - Reinstate Monica Apr 15 '15 at 16:39
  • @AleksandrBlekh and gung, I'm aware that the power can be written in terms of a non-central t-distribution, but I still didn't think it was possible to write down an explicit expression for the required sample size (or the power, for that matter) that can be evaluated by hand (with an ordinary calculator, not requiring R or similar software). I look forward to being proven wrong. – mark999 Apr 15 '15 at 19:31
  • @mark999, the formula has the normal CDF function $\Phi(\cdot)$ in it, which would require some kind of software or maybe a normal distribution table in the back of a stats book, if that's what you mean. But there is an analytical solution. – gung - Reinstate Monica Apr 15 '15 at 19:37
  • @gung That's part of what I meant (although I think it's a non-central t CDF, not a normal CDF), but I also meant that I think for the sample size, trial and error will be required (i.e., calculate the power for various sample sizes, and find the minimum sample size that gives power greater than or equal to the required power). Again, I hope to be proven wrong. – mark999 Apr 15 '15 at 20:21
  • @gung Can you post that formula please? – mark999 Jun 01 '15 at 03:30
  • @mark999, there is an analytical formula, but you couldn't solve it by hand, the roots are approximated numerically. I have given the formula for a z-test, and an iterative procedure to get you to a value. You are right that this will require some trial & error, but I think it would be doable w/ some perseverance & a sufficiently detailed t-table. – gung - Reinstate Monica Jun 01 '15 at 19:24
  • (+1) But "I will need 34 data" is an idiosyncratic usage even among those of us left who still treat "data" as grammatically plural. – Scortchi - Reinstate Monica Jun 01 '15 at 19:35
  • @gung In my opinion it's a bit of a stretch to say that there's an analytical formula, or an analytical solution, if it has to be solved numerically. Do you still claim that "G*Power is solving this analytically"? – mark999 Jun 02 '15 at 06:24
  • @mark999, I do. But perhaps we're using the terms differently. I don't think of numerical approximation of a function to be the same as simulating a system. There is an analytical formula that G*Power is solving, & it certainly isn't doing anything like what I discuss [here](http://stats.stackexchange.com/a/35994/7290). – gung - Reinstate Monica Jun 02 '15 at 14:23
  • @gung Yes we're using terms differently. I don't think of solving an equation numerically as being the same as using simulation either, but I find it strange (and misleading) that you would describe solving an equation numerically as "solving this analytically" and "there is an analytical solution". Thanks for the discussion. – mark999 Jun 03 '15 at 03:39
  • Note that "exact" confidence intervals are not very accurate. Also, what about the problem made hypothesis testing a good idea? Why was it not just an estimation problem for which the width of the confidence interval is the basis for designing the study? – Frank Harrell Oct 12 '21 at 12:00