9

I understand when performing a simple t-test, we typically control the type -1 error rate at $\alpha = .05$. This signifies that if the null hypothesis holds, the data will "incorrectly" reject the null hypothesis in 5% of all instances. Therefore, if we are performing 100 consecutive tests (and all of which the null hypothesis holds), we will reject the null in 5 of these tests. Thus, the need for multiple correction adjustment.

Currently, my statistics professor disagrees with my analysis. I will provide an example and then extrapolate further out:

  1. My data frame contains two groupings of people (say, type A and type B). For each person we have 100 different measurements (things ranging from: height, weight, IQ, etc...). I want to run 100 different t-tests comparing the means of each of my covariates among Type A and type B people. Am I still required to use a multiple correction scheme here? I say no because these are different measurements between groups A and B. They say yes because I am performing multiple tests.

  2. I happen to be the only statistician in my town. I am contracted to perform 100 different analyses (for 100 different researchers). Each researcher asks me to complete a single t-test. Do I need to perform a multiple test correction here? I would argue no though I fail to see the difference between this example and the previous.

Tommyixi
  • 203
  • 1
  • 5
  • An important difference is that in (1) there is _one_ sample of subjects, while in (2) there is a different sample of subjects for each of the researchers. – BruceET Oct 27 '19 at 18:26
  • 2
    This older thread was more focused on reproducibility, but might also be relevant: https://stats.stackexchange.com/questions/206592/why-arent-multiple-hypothesis-corrections-applied-to-all-experiments-since-the – Matt Krause Oct 27 '19 at 19:05

2 Answers2

10

Multiple comparisons corrections are intended to control the familywise error rate--or something like it--so they should be applied across a "family" of related hypothesis tests.

In your first example, the overarching goal probably to determine whether Groups A and B differ. If you didn't control for multiple comparisons, you could trivially find an effect by adding more and more tests: if there's no difference in height or weight, throw in BMI, Zodiac Sign, and annual ice cream consumption. Eventually, one of these tests will, by chance, cross your $\alpha$ threshold, and you'll be tempted to write something goofy like "Although As and Bs may seem similar, we observed a statistically significant difference (p=0.04) in the number of pets owned raised to the mother's age power".

In your second example, the tests are unrelated. Client #1 wants to know whether customers prefer red vs. orange trim on their cars. Client #2 wants to know if drug X shrinks tumors more than drug Y, Client #3 wants to know which factors affect her crop yields, and so on. These are all unrelated: any compound hypothesis relating them would be very bizarre.

So...how do you define "a family" of tests? I'm not sure that any hard-and-fast rules exist. Here are some guidelines:

  • Will multiple (formal) tests lead to the same substantial conclusion[*]? For example, suppose you're interested in the effects of diet on fitness. Differences in resting heart rate OR 40 yard dash time OR BMI OR bench press performance might lead you to the same conclusion: "Diet D improves fitness." If so, these tests all belong to the same family, and you ought to apply a multiple comparisons correction. T

  • Reusing the same subjects or the same hypothesis also suggests they belong to the same family. For example, maybe you test the distribution of salaries paid to men vs. women in 20 different countries. Each of those tests ought to be corrected.

[*] This requires a little subject matter expertise. For example, suppose you did 10 fitness tests, five of which could be said to measure "strength" and five of which measure "endurance." Do you apply a correction to all of the tests together, or separately within the "strength" and "endurance" families? Either might be okay, though you ought to be very clear about what you actually did. A better approach might be to generate composite scores, and sidestep the need for multiple comparisons corrections altogether.
Matt Krause
  • 19,089
  • 3
  • 60
  • 101
  • I think that's precisely the answer I was looking for; I am having a hard time identifying what a "family" of tests are. Though, this does make me wonder in the second example if we would still find 5 significant results even if the null hypothesis were true in each example and how we would change that? – Tommyixi Oct 27 '19 at 19:03
  • 1
    In the "consultant" example, the key is that no one cares whether "orange trim sells more cars OR drug X shrinks tumors better OR adding lime increases crop yields" is true, so there's no point in worrying about its error rate. In contrast, "drug X changes levels of protein A OR protein B or protein C" is a reasonable hypothesis, especially if those three proteins are all related somehow. – Matt Krause Oct 27 '19 at 19:18
  • 2
    This answer hit the nail on the head with respect to controlling the family-wise error rate. However, multiple testing corrections are not always intended to control this error rate, as the equally (if not more) popular *false discovery rate* is something different. – Frans Rodenburg Oct 28 '19 at 04:12
  • I interpreted OP's example 2 slightly differently: I thought it meant that the different researchers are asking for t-tests to be performed on the same sample of people (the townspeople). So the only difference between 1 and 2 would be that there are more researchers involved, but the individual t-tests are the same. – qdread Oct 28 '19 at 12:53
  • I think similar logic applies to FDR. In both cases, you're trying to control the "mistake"-rate over a group of related hypotheses; the thing that differs is what you're considering a mistake (the thread I linked to above talks about it, a little bit). – Matt Krause Oct 28 '19 at 13:11
  • -1 For not pointing out that the much more current *false discovery rate* methods (a) do not use a statistically undefined concept of "family", (b) which have rates of false rejections which scale across numbers of tests, and (c) have the added benefit of adaptively adjusting rejection probabilities based on accumulating evidence that *some* rejections are true rejections. Also "related tests" is bunk w/r/t the OP's question in the sense that *all* the tests share identical distributions of *p*-values under the null hypothesis. – Alexis Oct 28 '19 at 16:26
  • @Alexis, so how do *you* choose which p-values to pass into your FDR function? Presumably, it's because those p-values come from conceptually-related tests. No one controls their "lifetime FDR"; people don't even control FDR across separate experiments in a single paper. – Matt Krause Oct 28 '19 at 16:36
  • @MattKrause "people don't even control FDR across separate experiments in a single paper." People do. I am not saying that one does not make such a choice (I always choose all tests in a paper: literally every *p*-value presented), but that the "family" in FWER methods is not scalable: changing the number of tests changes the rejection probabilities, but this rejection probabilities are conserved in FDR methods: I can publish on 20 results, and in a further paper add 10 related-results into the pool and re-perform the multiple comparisons adjustment without expecting different results 1/2 – Alexis Oct 28 '19 at 23:16
  • @MattKrause 2/2 with respect to the first 20 tests' rejections. That is not true with the FWER methods. (E.g., if the second batch of tests were, say, all true rejections, FWER methods will tend to find previously significant results insignificant, but FDR methods will not.) – Alexis Oct 28 '19 at 23:17
  • 1
    Link? I have literally *never* seen someone do an FDR correction across an entire work. I have seen FDR applied to a list of candidate genes or brain areas that might change due to a disease. I'd be very surprised to see a subsequent test, like whether the called gene's expression levels correlates with disease severity, included in that correction. To me, that doesn't even make sense: 5% FDR in this figure, I can understand; I can't see when I'd want--or be able to use--a guaranteed FDR over unrelated tests. – Matt Krause Oct 29 '19 at 00:55
  • @MattKrause "I can't see when I'd want--or be able to use--a guaranteed FDR over unrelated tests." What do you know about the distribution of *p*-values under any null hypothesis? Can you help me understand what the benefit of rejection probabilities literally depending on the choice of *number of tests* is? – Alexis Oct 30 '19 at 01:01
  • I'm emphatically *not* arguing that FWER is better than FDR; I'm suggesting that both types of adjustment ought to be applied over groups of related tests so that the error/false discovery rates make sense. "No more than 5% of the genes I called last month and the EEG time-freq bins I found today are false discoveries" is a silly statement. "There are 20 hits on this list, and it's unlikely than more than 1-2 are false alarms" is not. Also FDR *is* sensitive--albeit less so--to the number of tests. Compare p.adjust(c(rep(0.01, 4), rep(1, N)), method='fdr') with N=15 and 17. – Matt Krause Oct 30 '19 at 13:05
1

The only real answer is - whenever you want to, as long as you can justify it. A reasonable case can be made for not adjusting at all. A reasonable case can be made for doing different things in different cases. This is more generally true of statistical analysis than most people think (see Statistics as Principled Argument by Robert Abelson for a lot more

First, recognize that 5% is arbitrary. Why not 1%? Why not 10%? Why is the usual power set at 80% or 90% instead of 95% or 99%?

Second, taking a tiny bit of a Bayesian view, how sure are we that the null is false? Except for some categorical tests with small populations, the null is never exactly true. If you (say) compare the heights of people with even and odd social security numbers and somehow get data for the entire population of the USA, the means will be different. Maybe in the 4th or 5th decimal place, but they won't be identical. But, in most research, we are pretty sure that the null isn't even approximately true.

The whole structure is an unfortunate legacy of Fisher's situation: He was testing different treatments of plots of land at Rothamstead. In this case, the notion that the null might be approximately true was sensible. Is treatment A better than treatment B? He had no real idea - so, he tested. But, often, we do know. If we are testing a medical treatment, for instance, we can be pretty sure it has some effect. The more certain we are, a priori that the null is not close to true, the less inclined we should be to adjust p values. Also, if recruiting more subjects and doing the measurements is expensive, we may think twice about adjusting. But if we have "big data" then it's no problem.

Third, remember that decreasing type I error will either increase type II error or require a bigger sample. Is a type I error worse than a type II error? The usual choices of 5% and 20% indicate that we think so. But sometimes type II error is much much worse. Suppose we have a drug that we think might reverse some disease which is terminal and currently untreatable. Then a type I error means giving an ineffective drug to a dying person while a type II error means letting someone die who could be cured.

Finally, remember that the whole significance testing apparatus is increasingly criticized, with varying degrees of severity. I'm a pretty harsh critic (you may have figured that out by now!) but who am I? Just some guy on Cross Validated. But the American Statistical Association has (to a much lesser degree) joined the critics.

Peter Flom
  • 94,055
  • 35
  • 143
  • 276