7

My dataset consists of $n$ genes, each of them described by a vector of expression values, $5$ for "healthy" individuals, and $5$ for "unhealthy" individuals.

I am going to run $n$ t-tests (one for each gene) to identify which genes show a different behaviour between healthy population and unhealthy population.

Should I consider a correction (such as Bonferroni, Holm, Benjamini & Hochberg...) for the $n$ p-values ?

EDIT:

I am wondering whether my case is a multiple comparisons problem or not.

Actually I do not compare the genes, but only the values of two different populations (healthy vs. unhealthy) for each gene. Therefore, I do not see the multiple comparisons.

In other words, I am interested in finding those genes that behave differently between healthy samples and unhealthy samples. I am not interested in finding whether or not two genes behave the same.

Obviously, running $n$ t-tests I get much more p-values lower that $0.05$ than after computing the correction.

no_name
  • 231
  • 1
  • 3
  • 7
  • 2
    Welcome! What is your existing understanding and where to things get confusing when you read the Wikipedia article on [controlling the false discovery rate](https://en.wikipedia.org/wiki/False_discovery_rate) or controlling the [family-wise error rate](https://en.wikipedia.org/wiki/Familywise_error_rate)? Is the the *how* (thankfully the how tends to be pretty easy, and often implemented in software), the *when* (you seem on the right track with $n$ comparisons), or the *why*? Or something else? Clicking the "edit" link in the lower left could let you clarify your question along these lines. – Alexis Dec 12 '14 at 19:38
  • Whether or not you apply corrections of some kind - and what kind of calculations you apply them to - depends on what properties you want your inference to have. In some situation you may not care about *any* corrections, if your type I error rate (or indeed your false discovery rate, if you care about that more) has been chosen in such a way that you already have the properties you want. You might for example say "the $\alpha$ I choose is in fact the Type I error rate I can live with on a per-comparison basis - why would I need to adjust?". (Note that dependence between tests may be possible) – Glen_b Dec 12 '14 at 21:20
  • @no_name: see also http://stats.stackexchange.com/questions/164181/family-wise-error-boundary-does-re-using-data-sets-on-different-studies-of-inde/164232#164232 –  Aug 16 '15 at 07:04

2 Answers2

6

You absolutely do want to apply a correction. The key idea is identifying significance by chance. As you increase the number of comparisons you increase the number of those that will be significant by chance.

For example, let's take the generic example of doing 100 comparisons using a significance threshold of 0.05. Now, a p-value of 0.05 means there is a 5% chance of getting that result when the null hypothesis is true. Therefore, if you do these 100 comparisons, you would expect to find 5 genes significant just by random chance.

As such, to avoid making these false-positives (Type 1 Errors) we 'correct' the p-value thereby making the test more conservative.

The choice in correction can vary too. Bonferroni is a common correction but if you have 1000's of genes, it is going to be exceedingly unlikely you will find anything significant because it will be so conservative. In that case, you may use the 'FDR' (False Discovery Rate) correction. There is no absolute answer so you need to explore the possibilities and make the best choice and of course report what correction you applied.

EDIT

Regarding you comments below I thought an example can help demonstrate the concept.

Using R, I generate completely random values for 250 genes with two treatments (A and B)

set.seed(8)
df <- data.frame(expression=runif(1000), 
                 gene=rep(paste("gene", seq(250)), 4), 
                 treatment = rep(c("A","A","B","B"), each=250))

I then split the data by each gene and run a t.test comparing between the two groups.

out <- do.call("rbind", 
    lapply(split(df, df$gene), function(x) t.test(expression~treatment, x)$p.value))

Now, given that this is completely random data there shouldn't be any significant differences and yet when I count how many there are 9 significant genes!!!

length(which(out < 0.05))
[1] 9

Avoiding mistakes like these is the point behind making these corrections. Hopefully this helps clarify for you.

cdeterman
  • 4,543
  • 1
  • 20
  • 34
  • I see your point, but in fact I am not doing comparisons at all. For each gene I have $5$ values corresponding to $5$ healthy individuals and $5$ values corresponding to $5$ unhealthy individuals. After the t-test I obtain a p-value for that gene, and I discard that gene if its p-value is greater than $0.05$. This procedure is computed for all the $n$ genes. Therefore, why should I correct the p-values if there are no comparisons between the genes? – no_name Dec 12 '14 at 20:29
  • I will add that there is an important philosophical difference embedded in FWER corrections vs. FDR corrections: the former assume that *no null hypotheses are false*, while the latter assumes that *some null hypotheses may be false*. The latter is, to the minds of many, more true to the spirit of scientific inquiry (i.e. we **do not** do science because we believe anything we study is unrelated to everything we study). – Alexis Dec 12 '14 at 20:29
  • 2
    @no_name Because *each time* you conduct the *t* test you are increasing the chance of a false positive. It's like the lottery: the more you play, the better your chance to "win" (where "win" is falsely rejecting a null hypothesis). – Alexis Dec 12 '14 at 20:31
  • But the computation of the p-value for gene 1 does not depend on the computation of the p-value for gene 1. There is no comparison between a pair of genes. – no_name Dec 12 '14 at 20:34
  • 2
    @no_name, please see my edit, what Alexis (whom I thank for the additional comments) is saying isn't that you are comparing between genes but you are **applying the same comparison across many genes**. The idea being, the more genes you measure the more likely you will find one significantly different by chance. – cdeterman Dec 12 '14 at 20:59
  • 1
    @no_name another way of looking at it: assuming $H_{0}$s are all true, what is your chance of making at least one Type I error when you have 1 test? Well, it is $\alpha$. What about when you have *two* tests? The probability, out of these two tests, of rejecting at least one $H_{0}$ is no longer $\alpha$, but (per the binomial distribution) $2\alpha(1-\alpha) + \alpha^{2}$... assuming $\alpha=0.05$ that probability equals 0.0975, almost *double* your chances than with a single test! – Alexis Dec 12 '14 at 21:07
  • @cdeterman: see also http://stats.stackexchange.com/questions/164181/family-wise-error-boundary-does-re-using-data-sets-on-different-studies-of-inde/164232#164232 –  Aug 16 '15 at 07:05
  • @Alexis: see also http://stats.stackexchange.com/questions/164181/family-wise-error-boundary-does-re-using-data-sets-on-different-studies-of-inde/164232#164232 –  Aug 16 '15 at 07:06
4

You say that no comparisons are being conducted because genes are not being compared to each other. However, each t-test is still a comparison. In fact, that's what a t-test is--a comparison of two means. In your case, each comparison is between the healthy group and the unhealthy group, rather than between gene A and gene B, but it is a comparison nonetheless.

This confusion can be avoided by substituting the synonym "multiple testing" for "multiple comparisons."

Bonferroni
  • 41
  • 1
  • 1
    Naturally Bonferroni knows about multiple testing. Multiple testing and multiple comparisons are synonymous. I am not sure what the confusion is but +1 for Bonferroni. – Michael R. Chernick Dec 30 '17 at 01:49