Questions tagged [statistical-power]

Is a property of a hypothesis testing method: the probability of rejecting the null hypothesis given that it is false, i.e. the probability of not making a type II error. The power of a test depends on sample size, effect size, and the significance ($\alpha$) level of the test.

The power of a statistical test $T_n$ is $\inf_{x\in H_1} P_x(T_n = 1)$ and it gives the probability of not making a type II error (rejecting the null hypothesis given that it is false). For simple hypothesis tests the $\inf$ reduces to a point.
Desirable properties for test statistics in general are low level and high power. Unfortunately in most procedures it is not possible to achieve both properties. For instance, whilst it is possible for the t-test to have low level and high power in a large sample setting, in the small sample case it can be shown that its level is $\alpha$ but that the power is at maximum $\alpha$ as well. This follows from the Karlin-Rubin theorem of uniformly most powerful tests.

For a fixed level and power it is possible to calculate the minimum effect size which is required to possibly identify a significant effect. This is known as the minimum detectable distance and it helps in assessing whether in an experiment or a regression it is at all possible to find a significant effect given a desired power.

1344 questions
90
votes
10 answers

Is there a minimum sample size required for the t-test to be valid?

I'm currently working on a quasi-experimental research paper. I only have a sample size of 15 due to low population within the chosen area and that only 15 fit my criteria. Is 15 the minimum sample size to compute for t-test and F-test? If so, where…
79
votes
7 answers

Rules of thumb for minimum sample size for multiple regression

Within the context of a research proposal in the social sciences, I was asked the following question: I have always gone by 100 + m (where m is the number of predictors) when determining minimum sample size for multiple regression. Is…
Jeromy Anglim
  • 42,044
  • 23
  • 146
  • 250
71
votes
5 answers

How can adding a 2nd IV make the 1st IV significant?

I have what is probably a simple question, but it is baffling me right now, so I am hoping you can help me out. I have a least squares regression model, with one independent variable and one dependent variable. The relationship is not significant.…
EvKohl
  • 1,090
  • 2
  • 10
  • 14
51
votes
4 answers

Cumming (2008) claims that distribution of p-values obtained in replications depends only on the original p-value. How can it be true?

I have been reading Geoff Cumming's 2008 paper Replication and $p$ Intervals: $p$ values predict the future only vaguely, but confidence intervals do much better [~200 citations in Google Scholar] -- and am confused by one of its central claims.…
amoeba
  • 93,463
  • 28
  • 275
  • 317
44
votes
5 answers

Why does increasing the sample size lower the (sampling) variance?

Big picture: I'm trying to understand how increasing the sample size increases the power of an experiment. My lecturer's slides explain this with a picture of 2 normal distributions, one for the null-hypothesis and one for the alternative-hypothesis…
user2740
  • 1,226
  • 2
  • 12
  • 19
42
votes
2 answers

Simulation of logistic regression power analysis - designed experiments

This question is in response to an answer given by @Greg Snow in regards to a question I asked concerning power analysis with logistic regression and SAS Proc GLMPOWER. If I am designing an experiment and will analze the results in a factorial…
B_Miner
  • 7,560
  • 20
  • 81
  • 144
38
votes
2 answers

Which statistical model is being used in the Pfizer study design for vaccine efficacy?

I know there's a similar question here: How to calculate 95% CI of vaccine with 90% efficacy? but it doesn't have an answer, at the moment. Also, my question is different: the other question asks how to compute VE, using functions from a R package.…
38
votes
3 answers

What percentage of a population needs a test in order to estimate prevalence of a disease? Say, COVID-19

A group of us got to discussing what percentage of a population needs to be tested for COVID-19 in order to estimate the true prevalence of the disease. It got complicated, and we ended the night (over zoom) arguing about signal detection and…
33
votes
5 answers

Why do political polls have such large sample sizes?

When I watch the news I've noticed that the Gallup polls for things like presidential elections have [I assume random] sample sizes of well over 1,000. From what I remember from college statistics was that a sample size of 30 was a "significantly…
samplesize999
  • 331
  • 3
  • 3
32
votes
6 answers

Sample size for logistic regression?

I want to make a logistic model from my survey data. It is a small survey of four residential colonies in which only 154 respondents were interviewed. My dependent variable is "satisfactory transition to work". I found that, of the 154 respondents,…
29
votes
3 answers

Sample size calculation for mixed models

I am wondering if there are any methods for calculating sample size in mixed models? I'm using lmer in R to fit the models (I have random slopes and intercepts).
28
votes
3 answers

Is it meaningful to test for normality with a very small sample size (e.g., n = 6)?

I have a sample size of 6. In such a case, does it make sense to test for normality using the Kolmogorov-Smirnov test? I used SPSS. I have a very small sample size because it takes time to get each. If it doesn't make sense, how many samples is…
27
votes
4 answers

Do underpowered studies have increased likelihood of false positives?

This question has been has asked before here and here but I don't think the answers address the question directly. Do underpowered studies have increased likelihood of false positives? Some news articles make this assertion. For example: Low…
25
votes
4 answers

Is power analysis necessary in Bayesian Statistics?

I've been researching the Bayesian take on classical statistics lately. After reading about the Bayes factor I've been left wondering if power analysis is a necessity in this view of statistics. My main reason for wondering this is the Bayes factor…
Justin Bozonier
  • 1,119
  • 2
  • 11
  • 24
25
votes
3 answers

Sanity check: how low can a p-value go?

I am using a ranksum test to compare the median of two samples ($n=120000$) and have found that they are significantly different with: p = 1.12E-207. Should I be suspicious of such a small $p$-value or should I attribute it to the high statistical…
N26
  • 1,705
  • 3
  • 18
  • 22
1
2 3
89 90