5

We know that underpowered statistics greatly increase the probability of a type II error (by definition), meaning a greater chance of failing to reject the null hypothesis despite the existence of a 'true' underlying effect.

The dangers of use of underpowered statistics are frequently taught to students in the context of the commonly used inferential statistics (i.e. ANOVA and t-test). However, I have never heard mention (including in that of statistics textbooks) of the importance of statistical power in relation to the supportive tests, such as those used to check for homogeneity of variances and normality of distribution.

I would argue that the dangers are even more apparent for these supportive tests, because there is a temptation to see a 'non-significant' p>.05 result and think "oh great, it's passed this step in the process!" and assume normality and homoscedasticity. However, surely if our Levene's test is underpowered, there will be a high probability of obtaining a 'non-significant' result even if, for example, the true variances are not equal.

In short, if our supportive statistics are underpowered, we cannot assume that the assumptions for ANOVA, t-tests, and others have been met solely on the basis of a 'non-significant' result.

My questions are: - Am I correct in assuming statistical power is important for the supportive tests?

  • What should we do about it? Never perform statistics on low samples? Cross-reference non-significant results on supportive tests with plots and visuals?
Josh Blake
  • 73
  • 5

1 Answers1

1

Perhaps better as a comment (I don't have concrete conclusions), but long ...

I think the most important thing to keep in mind is that the question we are trying answer is not

do the data violate the assumptions of the model we are using?

(we know they do: outside of simulations, and possibly some areas in physics, nothing is ever really perfectly Gaussian/linear/homoscedastic/independent ...)

but rather

are the violations severe enough that they compromise our results?

This difference is why we don't care about rejecting the null hypothesis of e.g. Normality when our data set is large/"overpowered"; in this cases the model conclusions are usually fine even though we can detect violations of the assumptions.

The question, then, is "when we have small data sets, how often will we fail to detect violations of assumptions that are large enough to be problematic? (i.e. lead to large biases, inflation of false positive rates above nominal, etc.)" ?

Ben Bolker
  • 34,308
  • 2
  • 93
  • 126
  • I completely agree. It is not just about the size of the effect (i.e. deviation from normality), but the resulting severity of violation. I imagine these two do not perfectly correlate. Hard and fast rules should, of course, never be applied dogmatically, but I was wondering how you judge 'meaningful differences' in both the cases of underpowered and overpowered samples? As a relative layman in the world of statistics, it seems hard to ascertain whether statistically significant findings regarding normality and homogeneity of variances are actually statistically **meaningful**. – Josh Blake Jan 25 '19 at 15:37