1

although I have studied psychology and feel I should know a lot more about issues such as this one, I have a quite basic question (or maybe not so basic question, considering all the fuss about this) concerning multiple testing.

The first, and back then very confusing contact with things such as Bonferroni, Bonferroni-Holm, or Dunn-Sidak corrections was six years ago, when I started taking statistics courses at university. Multiple testing issues were always applied to post-hoc tests in ANOVA, or t-tests. Yet, it appears obvious to me that such corrections are necessary in many more situations. However, there appears to be little consensus about if, and if yes which correction to use.

I have read in various threads, e.g. here, here, or here that somehow the degree of (in)dependence of the multiple tests influences how striking the problem of multiple testing is. This, in some sense, appears obvious to me, yet I do not understand lines of (mathematical) reasoning behind this issue.

Furthermore, I guess the issue of (in)dependence directly influences whether or not or how to correct alpha, since I guess two t-tests with different variables and different groups, although part of the same research question are less dependent than, say, two t-tests on different variables but with the same groups, or even some multiple regression within which the significance of multiple regression weights is tested.

I hope this question is of a more general nature and might be of interest to more people but myself. So in essence:

Could anyone provide me with a (not too mathematically elaborate, I am not a statistician) explanation in which way (in)dependence of statistical tests influences how effectively problematic multiple testing is? (If there are useful links to articles/websites/questions & answers that are of more mathematical nature, please provide a link. There might be others more keen & able to understand the mathematics behind.)

I hope that answers to this question provide me with more heuristic knowledge to choose an appropriate correction when performing multiple tests. Thanks in advance!

Community
  • 1
bunsenbaer
  • 177
  • 2
  • 14

1 Answers1

1

I personally think multiple testing is not nearly as big of a deal as people make it out to be. I just accept the fact that as more tests are done more mistakes are going to be made.

But, if your goal is to control a family-wise error rate then it's useful to recall some results from probability. For a collection of events $\{A_1, A_2, \ldots, A_n \}$, the probability that at least one of these events occurs is given by the inclusion-exclusion formula,

$$ \begin{align} P(A_1 \cup A_2 \cup \ldots \cup A_n) &= \sum_{i=1}^{n} P(A_i) - \underset{i \neq j}{\sum \sum} P(A_i \cap A_j) + \\ & \underset{i \neq j, j \neq k, i \neq k}{\sum \sum \sum} P(A_i \cap A_j \cap A_k) - \ldots + (-1)^{n + 1} P(\cap_{i=1}^{n} A_i) . \end{align} $$

Without knowledge of the probabilities of these intersections (i.e., the dependencies between the events) we can bound this using Boole's inequality,

\begin{align} P(A_1 \cup A_2 \cup \ldots \cup A_n) &\leq \sum_{i=1}^{n} P(A_i) . \end{align}

This is the idea behind the Bonferroni method, to control the probability of at least one error by addressing the worst case when the errors are actually mutually exclusive events. This situation is the most problematic scenario for multiple testing, but it's also highly unrealistic. Unfortunately, it's practically impossible to know much about how mistakes might be correlated when performing multiple hypothesis tests, which is why some have still used this method.

There's an alternative approach to these problems which doesn't try to control family-wise error rate (in certain large scale studies this might be unfeasible), but rather something called a "false discovery rate": https://en.wikipedia.org/wiki/False_discovery_rate

dsaxton
  • 11,397
  • 1
  • 23
  • 45
  • Boole's inequality is a direct consequence of the additivity axiom of probability: it does not derive from the inclusion-exclusion formula. One wonders, then, what light that formula sheds on this situation. Isn't the analysis made much easier by considering the complementary event where *none* of the events occurs? Understanding Bonferroni comes down to understanding *independence* rather than mutual exclusion. – whuber Jul 10 '15 at 20:55
  • I didn't say Boole's inequality comes from the inclusion-exclusion formula. The formula is useful because it shows exactly how the family wise error rate is affected by the dependence structure of the events involved. Seems appropriate if understanding independence is the important point. But sure, I guess you could rewrite it as $1 - P(\cap_{i=1}^{n} A_n^c)$ if you want. – dsaxton Jul 10 '15 at 21:11
  • The key point is the distinction between independence and mutual exclusion. By focusing on mutual exclusion, which doesn't seem to apply to any Bonferroni application, you seem to be leading the OP and your readers away from any insightful explanation. – whuber Jul 10 '15 at 21:13
  • The poster asked how problematic dependence can be in multiple testing. It's most problematic when the errors are mutually exclusive events because this maximizes the family wise error rate. The idea obviously applies to Bonferroni correction because it's the only time when the bound used is actually achieved. – dsaxton Jul 11 '15 at 00:19
  • I guess I'm struggling to understand how a collection of hypothesis tests would have mutually exclusive outcomes: a failure of the imagination, perhaps. Would you mind indicating in what circumstances this occurs? – whuber Jul 13 '15 at 19:27