Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [calibration]

Calibration can refer to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc. Do not use for predicting an explanatory variable from an observation of the dependent variable, for that use the tag inverse-prediction.

In regression, calibration can refer to predicting an explanatory variable from an observation of the dependent variable or to adjustment of measurements to agree with value of some standard.

In machine learning it may refer to transforming classifier scores into class membership probabilities.

It can also refer to performing inference on model parameters.

247 questions
23
votes
2 answers

Scikit correct way to calibrate classifiers with CalibratedClassifierCV

Scikit has CalibratedClassifierCV, which allows us to calibrate our models on a particular X, y pair. It also states clearly that data for fitting the classifier and for calibrating it must be disjoint. If they must be disjoint, is it legitimate to…
sapo_cosmico
  • 374
  • 1
  • 2
  • 10
23
votes
3 answers

Visualizing the calibration of predicted probability of a model

Suppose I have a predictive model that produces, for each instance, a probability for each class. Now I recognize that there are many ways to evaluate such a model if I want to use those probabilities for classification (precision, recall, etc.). …
Michael McGowan
  • 4,561
  • 3
  • 31
  • 46
19
votes
2 answers

Why use Platt's scaling?

In order to calibrate a confidence level to a probability in supervised learning (say to map the confidence from an SVM or a decision tree using oversampled data) one method is to use Platt's Scaling (e.g., Obtaining Calibrated Probabilities from…
B_Miner
  • 7,560
  • 20
  • 81
  • 144
15
votes
1 answer

Pooling calibration plots after multiple imputation

I would like advice on pooling the calibration plots/statistics after multiple imputation. In the setting of developing statistical models in order to predict a future event (e.g. using data from hospital records to predict post hospital discharge…
IWS
  • 2,554
  • 13
  • 30
14
votes
2 answers

How to choose optimal bin width while calibrating probability models?

Background: There are some great questions/answers here on how to calibrate models which predict probabilities of an outcome happening. For example Brier score, and its decomposition into resolution, uncertainty and reliability. Calibration plots…
Alex
  • 3,728
  • 3
  • 25
  • 46
11
votes
3 answers

How do I choose the best metric to measure my calibration?

I program and do test-driven development. After I made a change in my code I run my tests. Sometimes they succeed and sometimes they fail. Before I run a test I write down a number from 0.01 to 0.99 for my credence that the test will succeed. I…
Christian
  • 989
  • 1
  • 9
  • 28
10
votes
5 answers

What is calibration?

What does it mean to calibrate survey weights? Also, what are other definitions of calibration in statistics? I have heard it used in several contexts, particularly risk prediction (referring to whether the total number of predicted events in a…
AdamO
  • 52,330
  • 5
  • 104
  • 209
9
votes
2 answers

Why is logistic regression well calibrated, and how to ruin its calibration?

In the scikit learn documents on probability calibration they compare logistic regression with other methods and remark that random forest is less well calibrated than logistic regression. Why is logistic regression well calibrated? How could one…
sjw
  • 5,091
  • 1
  • 21
  • 45
9
votes
3 answers

Probability calibration from LightGBM model with class imbalance

I've made a binary classification model using LightGBM. The dataset was fairly imbalnced but I'm happy enough with the output of it but am unsure how to properly calibrate the output probabilities. The baseline score of the model from…
Auren Ferguson
  • 221
  • 2
  • 3
9
votes
1 answer

How to estimate a calibration curve with bootstrap (R)

Question: I have fitted a probabilistic model (bayesian network) for modeling a binary outcome variable. I would like to create a high-resolution calibration plot (e.g. spline) corrected for overfitting with bootstrapping. Is there a standard…
Gino_JrDataScientist
  • 437
  • 2
  • 12
9
votes
4 answers

Tuning an exponential moving average to a moving window mean?

The alpha parameter of an exponential moving average defines the smoothing that the average applies to a time series. In a similar way, the window size of a moving window mean also defines the smoothing. Is there some way to tune the alpha parameter…
naught101
  • 4,973
  • 1
  • 51
  • 85
9
votes
1 answer

How to make calibration plot for survival data without binning data?

To make a calibration plot for survival probabilities estimated from a Cox model, one can divide the estimated risk into groups, calculate the average risk within a group, and then compare this to the Kaplan-Meier estimate. What are alternative…
julieth
  • 2,252
  • 1
  • 15
  • 20
8
votes
2 answers

Calibration of Cox regression survival analysis

To perform calibration of a Cox regression model (i.e. assessing for the agreement between the predicted and the observed outcome), what is the best method to present the accuracy of the model in predicting the actual event? As far as I…
user32454
  • 131
  • 2
  • 6
8
votes
1 answer

Signatures of underfitting and overfitting in logistic regression calibration curves

My confusion stems from reading the following paper http://www.bmj.com/content/351/bmj.h3868 It states in its abstract (and they later show an empirical study that conforms to the claim) - "Overfitted models tend to underestimate the probability of…
Ayush Saxena
  • 81
  • 4
7
votes
0 answers

Calibration for random forests

I want to evaluate the calibration of the random forest using val.prob (rms package, R). I have no problems using it and getting an output, but I feel the results may not be accurate because I don't believe that the class membership probabilities…
user4673
  • 1,461
  • 4
  • 17
  • 26
1
2 3
…
16 17