5

I want to calculate confidence intervals of positive likelihood ratios of multiple dependent tests.

To adjust the confidence level for the 'problem of multiple comparisons', I believe I should control the False Coverage Rate for the confidence intervals, using the Benjamini–Hochberg procedure (or something equivalent) to take into account the positive dependence of the tests.

However, it is not entirely clear to me how to do this for confidence intervals of positive (or negative) likelihood ratios, since the Benjamini–Hochberg procedure requires p-values. Is there a way to convert positive likelihood ratios to p-values or should I do something else?

nominator
  • 237
  • 1
  • 5
  • 12

1 Answers1

2

The idea behind False-Coverage-rate (FCR) adjusted confidence intervals (CIs) is that if you can construct regular CIs, you merely need to inflate them for FCR control. The inflation is done by constructing the marginal CIs to the level $\alpha R/m$ where $\alpha$ is the desired FCR level, $R$ is the number of selected parameters, and $m$ is the number of candidate parameters.

If you are constructing intervals on all considered parameters (i.e., $R=m$), there is no need for correction. Regular intervals will control the FCR [1].

For an application, and brief intro to other FCR controlling methods, see [2].

[1] Benjamini, Y., and D. Yekutieli. “False Discovery Rate-Adjusted Multiple Confidence Intervals for Selected Parameters.” Journal of the American Statistical Association 100, no. 469 (2005): 71–81.

[2] Rosenblatt, J. D., and Y. Benjamini. “Selective Correlations; Not Voodoo.” NeuroImage 103 (December 2014): 401–10. doi:10.1016/j.neuroimage.2014.08.023.

JohnRos
  • 5,336
  • 26
  • 56
  • 1
    I don't understand "If you are constructing intervals on all considered parameters, there is no need for correction. Regular intervals will control the FCR". Why would there be no need for correction in this case? (I couldn't find this in the referred article). – nominator Apr 26 '15 at 09:26
  • 1
    Also, it's still not clear to me how to select parameters (if R – nominator Apr 26 '15 at 09:28
  • 1
    From [1]: unadjusted CIs seem more acceptable than "... they give the right coverage on average; the proportion of 95% CIs covering their respective parameters out of the intervals constructed (namely, the number covering divided by the number of parameters m) is expected to be .95, and thus only .05 will not be covered..." – JohnRos Apr 26 '15 at 12:12
  • Regarding the selection: Are you constructing intervals for all likelihood ratios? If so, there is no selection (R=m). How to construct a marginal interval on the likelihood ratio is not a multiple-testing matter. Just construct the intervals as you would construct for inference on a single parameter. – JohnRos Apr 26 '15 at 12:15
  • Basically I want to select the parameters that have high positive likelihood ratios ("significantly" above 1) with a certain level of confidence. That's why I am calculating confidence intervals for them. It seems like a multiple-testing matter to me, because there is definitely going to be selection. – nominator Apr 26 '15 at 14:05
  • Good. Then the number of likelihood ratios you select is R. – JohnRos Apr 26 '15 at 14:06
  • But how do I calculate R? That is the main question. – nominator Apr 26 '15 at 14:08
  • For the purpose of FCR control, it does not matter. You can use hypothesis testing, or any arbitrary cutoff. – JohnRos Apr 26 '15 at 14:39
  • So could I just select the likelihood ratios that are >= 2 (for example) and then construct 1 - Rq/m confidence intervals for those? – nominator Apr 27 '15 at 20:04
  • Absolutely. See the Methods/Overview subsection in [2]. – JohnRos Apr 28 '15 at 06:04
  • 2
    Actually the problem I have with this approach is that I want to select parameters based on their significance. The number of parameters to select should not be fixed a priori, and not just based on the likelihood ratios because that would ignore their significance. That's why I prefered the "FCR-Adjusted BH-Selected CIs" procedure. It selects parameters based on their p-values and then constructs 1-Rq/m CIs for them. The only problem I had, as stated in my question, was how to get p-values from likelihood ratios. But I have figured this out now. – nominator Apr 30 '15 at 15:09
  • I came to this question while searching for an answer to a similar question I posted to SE. Just to get this right @JohnRos, if I construct five confidence intervals and 3 of them are significant at the 0.05 level (i.e., their 95% confidence intervals don't include zero), do I need to construct their new confidence intervals at the /=0.05*3/5=0.03 level (which is 97% interval)? My original question is here: https://stats.stackexchange.com/questions/501346/multiple-comparison-correction-for-confidence-interval – vbip May 20 '21 at 21:04