Questions tagged [optimism]
6 questions
2
votes
1 answer
Estimating regression optimism using the bootstrap
I am estimating optimism bias in for example risk predictions. A method for doing that is described by Frank Harrell and implemented in the R package rms. I am trying to reproduce the bias-corrected calibration curve in SAS. However I am getting…
Danny
- 805
- 5
- 17
2
votes
0 answers
Optimism bootstrap with non-linear models
I have come across an example in my research with heavily overfit non-linear probabilistic classifiers, where the optimism bootstrap appears to underestimate the optimism, even when using a proper scoring metric (log-likelihood/cross-entropy loss).…
thebigspin
- 31
- 3
2
votes
1 answer
In which scenarios are the in-sample error and training error NOT the same?
In Elements of Statistical Learning, Chapter 7 (pages 228-229), the authors define the optimism of the training error rate as:
$$
op\equiv Err_{in}-\overline{err}
$$
With the training error $\overline{err}$ defined as:
$$
\overline{err} =…
Skander H.
- 10,602
- 2
- 33
- 81
1
vote
0 answers
What is a reliable way to obtain an optimism-correct AUC with confidence limits?
I have seen that Frank Harrell's rms package does not offer a CI for Somers Dxy (and subsequently the c-statistic/AUROC).
I am trying to look at a method with good performance and a plan to implement this properly, probably through R.
I know an…
LSC
- 734
- 3
- 10
0
votes
1 answer
Expected Optimism 0-1 Loss with 0-1 Response
Want to show that
$$ E_X op = \frac{2}{n} \sum_{i=1}^n Cov_X(g(x_i), Y_i)$$
For 0-1 loss function with 0-1 response.
Want I've done
$$op = l_{in}(g) - l(g)=\frac{1}{n}\sum_{i=1} ^n Loss(Y_i', g(x_i ))-\frac{1}{n}\sum_{i=1} ^n Loss(Y_i, g(x_i ))$$…
CoderOgden
- 101
- 1
0
votes
2 answers
Statistical evidence that the AUC was not overfitted to the model. With N=119, C-stat = 0.81 seems optimistic. Optimism-adjusted?
My data have 119 cases and we did ROC for x (continuous variable) to predict postoperative y (categorical variable) available here, we got a comment from a reviewer asking:
Please provide statistical evidence that the AUC was not overfitted to…
Mohamed Rahouma
- 151
- 5