How to estimate False Discovery Rate from p-value distribution?

Question

I have learned many models and I calculated p-values for the cross-validation errors. I want to select significant models based on the false discovery rate (FDR). How can I estimate the FDR from p-value distribution?

Answer 1

Generally speaking, the false discovery rate correction requires ranking your p-values and then computing a critical value (based on the false discovery rate) that you can compare to your p-values. This website gives a very concrete example: here.

If you're using R, then it's even easier to do the adjustment. The following code is a quick example:

pvals <- c(0.04, 0.03, 0.06, 0.01, 0.02, 0.003)
p.adjust(p = pvals, method = "BH")

In the above code, the object pvals is a list of the p-values that you obtained from your analyses. The function p.adjust provides several different options for adjusting p-values based on multiple observations. In this case, the false discovery rate adjustment can be done by specifying method = "BH" for Benjamin-Hochberg or method = "fdr" for false discovery rate (note: they are the exact same procedure and will give the exact same results).