0

I am trying to generate 95% confidence intervals for the accuracy predictions on the trained data set using the caret package and interface.

Some dummy code:

library(earth)

Set control parameters

Tcontrolparam.Acc = trainControl("cv", savePredictions = TRUE,  classProbs = T, returnResamp = "final")

Set output to factor - classification - dw about names - just dummy data

trees$class = c(rep(c("John","Steve"),nrow(trees)/2),"Steve")

Model 1 using glm model

set.seed(0)
x = train(x = select(trees, -class), y = trees$class,
                                  method = "glm",
                                  metric = "Accuracy",
                                  trControl = Tcontrolparam.Acc)

Model 2 using MARS (earth) model

y = train(x = select(trees, -class), y = trees$class,
          method = "earth",
          metric = "Accuracy",
          trControl = Tcontrolparam.Acc)

Combining models into list and generating resampled results

Model.Objects.List = list(x,y)

Results = resamples(Model.Objects.List)

Summary.Results = summary(Results)

The code above and specifically Summary.Results gives a mean value for accuracy across the resamples but not the confidence intervals. Is there an easy way of computing confidence intervals using caret or externally across many models?

JFG123
  • 133
  • 7
  • Accuracy is doubtful as a measure of your predictive power in the first place: [Why is accuracy not the best measure for assessing classification models?](https://stats.stackexchange.com/q/312780/1352) and [Is accuracy an improper scoring rule in a binary classification setting?](https://stats.stackexchange.com/q/359909/1352) – Stephan Kolassa Jul 31 '18 at 14:23
  • @StephanKolassa HI Stephan, if I was to use ROC to tune my model I still do not know how to get the confidence intervals in this user interface. – JFG123 Jul 31 '18 at 15:43

0 Answers0