0

My data has 119 case and We did ROC for x (continuous variable ) to predict postoperative y (categorical variable) available here, we got a comment from a reviewer asking "Please provide statistical evidence that the AUC was not overfitted to the model. With N=119, C-stat = 0.81 seems optimistic. Optimism-adjusted? "

I found some answers; basically I split my cohort into train set (n=107) and test set (n=12), did 2 ROC curves and compared AUC for both using this command:

testobj <- roc.test(rocobj0, rocobj1);testobj 

and this was the output

DeLong's test for two ROC curves

data: rocobj0 and rocobj1

D = 0.44822, df = 12.488, p-value = 0.6617

alternative hypothesis: true difference in AUC is not equal to 0

sample estimates:

AUC of roc1 AUC of roc2

80.26961 72.22222

Regarding optimism adjustment, I did it used the following codes obtained from here

auc.adjust <- function(data, fit, B){
  fit.model <- fit
  data$pred.prob <- fitted(fit.model)
  auc.app <- roc(data[,1], data$pred.prob, data=data)$auc # require 'pROC'
  auc.boot <- vector (mode = "numeric", length = B)
  auc.orig <- vector (mode = "numeric", length = B)
  o <- vector (mode = "numeric", length = B)
  for(i in 1:B){    
    boot.sample <- sample.rows(data, nrow(data), replace=TRUE) # require 'kimisc'
    fit.boot <- glm(formula(fit.model), data = boot.sample, family = "binomial")
    boot.sample$pred.prob <- fitted(fit.boot)
    auc.boot[i] <- roc(boot.sample[,1], boot.sample$pred.prob, data=boot.sample)$auc
    data$pred.prob.back <- predict.glm(fit.boot, newdata=data, type="response")
    auc.orig[i] <- roc(data[,1], data$pred.prob.back, data=data)$auc
    o[i] <- auc.boot[i] - auc.orig[i]
  }
  auc.adj <- auc.app - (sum(o)/B)
  boxplot(auc.boot, auc.orig, names=c("auc.boot", "auc.orig"))
  title(main=paste("Optimism-adjusted AUC", "\nn of bootstrap resamples:", B), sub=paste("auc.app (blue line)=", round(auc.app, digits=4),"\nadj.auc (red line)=", round(auc.adj, digits=4)), cex.sub=0.8)
  abline(h=auc.app, col="blue", lty=2)
  abline(h=auc.adj, col="red", lty=3)
}

model <- glm(data$y ~  data$x, data = data, family = "binomial")
auc.adjust(df, model, B=200)

But it gave me this weird figure in which AUC is > 1

img

So A) I should conclude that AUC was not overfitted based on the splitting of the data that I did and P value of 0.66 from DeLong's test for two ROC curves, Right?

B) How to fix the optimism adjusted AUC figure to get it <=1?

0 Answers0