clustered multi-reader multi-case/modality ROC analysis in R

Question

My problem

I am looking for an R approach to:

do a ROC-Analysis of (clustered (i.e. several observations per case, e.g. prostate segments) / multi reader / multi case/motality) rating (score 1-5) data and compare AUCs of different readers and modalities.
calculate Sensitivity and Specificity (with CIs) of the identical, but binarized rating data.

An example of my data looks like:

library("dplyr")

df <- tibble(caseID=rep(1:5,each=8),
        modalityID=rep(c("A","A","B","B"), 10),
        segmentID=rep(c("a","b"),20),
        readerID=rep(rep(1:2, each=4),5),
        score=sample(1:5, 40, replace=TRUE),
        truth=sample(0:1, 40, replace=TRUE))

df <-df %>% mutate(scoreBIN=case_when(score>=3 ~1, TRUE ~ 0))

df


# A tibble: 40 x 7
   caseID modalityID segmentID readerID score truth scoreBIN
    <int> <chr>      <chr>        <int> <int> <int>    <dbl>
 1      1 A          a                1     2     1        0
 2      1 A          b                1     3     1        1
 3      1 B          a                1     1     1        0
 4      1 B          b                1     4     1        1
 5      1 A          a                2     5     0        1
 6      1 A          b                2     1     0        0
 7      1 B          a                2     4     1        1
 8      1 B          b                2     5     0        1
 9      2 A          a                1     5     0        1
10      2 A          b                1     2     0        0
# ... with 30 more rows

My approach

I found an example in R: http://www.lerner.ccf.org/qhs/software/lib/clusteredROC_help.pdf on https://www.lerner.ccf.org/qhs/software/roc_analysis.php that considers the clustered nature of my data, but not the multi modality.

I found a very handy R package (iMRMC) that offers multi-reader, multi-case analyses: https://cran.r-project.org/web/packages/iMRMC/index.html. However, this package cannot treat clustered data yet.

I've learned that I could use a hierarchical bootstrap approach (as described here https://www.tandfonline.com/doi/abs/10.1080/03610920802610084) and bootstrap over cases to take out all observations from the cases.

I found this tutorial to bootstrap clustered data: https://www.r-bloggers.com/bootstrapping-clustered-data/

What I do not know

How to implement a hierarchical bootstrap approach and get a data set that I could use with the iMRMC package
How to correctly calculate Sensitivity and Specificity (and CIS) of the binarized ratings.

Update:

I refined this question here: hierarchical bootstrapping and calculation of variance (in a Random-Effects ROC Analysis) in R

clustered multi-reader multi-case/modality ROC analysis in R

0 Answers0