0

I was wondering which method among "fdr","none","bonferroni" is the most accurate one to calculate the adjusted p-value for multiple comparison tests. Since when I define

summary(glht(fit1.2,linfct=mcp(Ethnic.Sex=cMat)),test=adjusted("none"))

I will have several significant p-values, but when I use Bonferroni and fdr the p-values are not significant. So any advice would be highly appreciated?

Ferdi
  • 4,882
  • 7
  • 42
  • 62
joe
  • 87
  • 1
  • 10
  • 2
    Bonferroni is very conservative. FDR deals how many errors are allowed, a different criterion compared to maximizing the probability of no errors. – Michael R. Chernick Aug 15 '17 at 14:44
  • 2
    As @Michael Chernick argues these procedures have different goals: Bonferonni controls the Familiy Wise Error Rate (FWER) and FDR controls False Discovery Rate (FDR). –  Aug 15 '17 at 15:02
  • It might help if you could include in your question the number of comparisons you are performing and provide some idea of the scale of your data set. – EdM Aug 15 '17 at 15:29

1 Answers1

0

The idea behind these corrections is that the the chance of getting a false positive for several test say 20 at an $\alpha=0.05$ is 1 in 20. Out of 20 comparisons one has a strong possibility of being a false positive.

"none" says don't correct the test and use $\alpha = 0.05$.

The Bonferroni correction is the most conservative measure of the three you mentioned where the adjusted significance level is $\alpha/m$ where $m$ is the number of hypotheses.

FDR or False Discovery Rate correction in R is an alias for the 'BH' Benjamini & Hochberg correction. This method ranks the unadjusted p-values and the new pvalue is less than or equal to $\alpha * rank / m$ where $m$ is the number of hypotheses.

As a rule of thumb, in terms of a false positive I would order them as follows (lowest to highest):

bfr $<$ fdr $<$ none

You can do more research into each and learn about the family wise error rate vs the false discovery rate.

Question and answer on Stackexchange about the bonferroni correction vs Benjamini-Hochberg correction as the number of comparisons increase

Matt L.
  • 739
  • 3
  • 10
  • thanks so much for your great advice. But does it really make sense to not adjust the p value (none) when we have several hypotheses to test? For example, I have two-way ANCOVA and I want to do the multiple comparison tests between levels. Do you think, I should use "none" and not adjust the p-value? – joe Aug 15 '17 at 15:14
  • Also, why adjusted Bonferroni p-values are usually 1? – joe Aug 15 '17 at 15:22
  • How many comparisons are being made? If there are many factors $m$ could be large and the significance level would be very conservative which would result in high p-values for small differences. I wouldn't go with none if the Bonferroni looks too conservative I'd switch to BH (fdr) correction method. – Matt L. Aug 15 '17 at 15:54
  • @ Matt L., thanks so much, I have a variable with 2 levels and the second variables with 4 levels. So it would be around 28 comparisons. – joe Aug 15 '17 at 16:02
  • I think you forgot some rather important conditionals on your sta – Björn Aug 15 '17 at 17:37
  • ments in the answer – Björn Aug 15 '17 at 17:38
  • @ Björn, Thanks for the reply. important conditionals? – joe Aug 16 '17 at 02:33