5

I'm working on classifying models for a few different projects. Several papers on the subject of calibration all suggest using isotonic regression (using PAV) to adjust the model probabilities.

I like the proposed calibration step, but am unsure how to apply it to NEW predictions from the model It appears as if the tools in both R an Python will happily calibrate probabilities if you also provide the true labels.

How can I then apply this to new data where the true labels are unknown.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
user21067
  • 61
  • 3

1 Answers1

3

The result of isotonic regression is a function that is fitted to the data you provide. You should train the isotonic regression just like you would train the underlying model (using some sort of out of sample data to prevent bias). You would apply the function obtained by fitting the isotonic regression model (using PAV) to the probabilistic output of your base model.

If you're using the Iso package in R, you need to use specify stepfun = T as an argument to pava(). Example:

library(Iso)
y <- (1:20) + rnorm(20)
f <- pava(y, stepfun = TRUE)
x <- runif(1)
f(x) # returns calibrated value

See also: Using Platt Scaling and Isotonic Regression to Minimize LogLoss Error in R

Nicholas Normandin
  • 51
  • 7