0

I am building an accelerated time failure (AFT) model on a highly imbalanced data set 90% survival 10% death. I understand that we can not use Brier score because of the outcome imbalance, Brier score dose not fair well with imbalanced data, and I am looking at the C-index and D-calibration to help validate my model. I am struggling to understand how to interpret the D-calibration score. The following paper is the best resource I could find explaining D-calibration but I am still confused on interpreting the outcome. If the p value is close to significant, e.g., 0.052, what is that saying?

Effective Ways to Build and Evaluate Individual Survival Distributions pg.16 https://arxiv.org/pdf/1811.11347.pdf

Using this package to calculate the D-Calibration with the following output options

  • statistic returns the chi squared test statistic
  • pval returns the chi squared test p value
  • max_deviation returns the maximum percentage deviation from the expected value, calculated as abs(expected_percentage - real_percentage), where expected_percentage = 1.0/n_bins
  • histogram returns the full calibration histogram per bin
  • all returns all of the above in a dictionary

https://loft-br.github.io/xgboost-survival-embeddings/modules/metrics.html

user14473305
  • 1
  • 1
  • What’s wrong with Brier score? // What about log loss? – Dave Mar 30 '21 at 02:35
  • Brier score does can not handle imbalanced data outcome from my understanding. "A noted shortcoming of the Brier score is that it doesn’t fare well when the classes are imbalanced." https://statisticaloddsandends.wordpress.com/2019/12/29/what-is-a-brier-score/ – user14473305 Mar 31 '21 at 13:36
  • Is there some text that backs up the use of log loss for survival models – user14473305 Mar 31 '21 at 13:41
  • Please say more about the nature of your data and your analysis: how many cases total, how many predictors you are evaluating, why you are choosing an AFT model, what type of AFT model your are building, why you are using `xgboost`, how you intend to use your model, whether you are interested specifically in trying to estimate individual survival curves (as in the linked paper) or if population-level analysis is adequate, etc. The best way to proceed might depend on that information. Please edit your question to include that information, as comments can be easily overlooked or deleted. – EdM Apr 01 '21 at 13:57

0 Answers0