1

I'm looking for alternatives to RPS to evaluate if my model is "good", as I can't use the RPS (because I do not have the actual probabilities).

My problem is a multiclass classification (30 categories) that something will happen in 1 day, in 2 days, in 3 days, ..., in 30 days. I'm treating this as a classification problem because I need to predict probabilites (and many of the estimators from sklearn has predict_prob method.

As I do not have the actual probabilities, I'm trying to find a measure that can be calculated using the actual class (instead of actual probability).

I was thinking of Brier Score (for multiclass) due this question: How to Compute the Brier Score for more than Two Classes

Is Brier Score good for evaluate my models and then choose the "best model" from Bier Score and suppose that this model would generate the lower RPS? Or should I use another metric?

Obs.: the classes are imbalanced (the class 30 has approximately 40% of frequency against others classes).

igorkf
  • 269
  • 2
  • 10
  • 1
    There is an ordering to your outcomes, which the multiclass Brier and the log score would not capture. You should be able to just use the RPS with deterministic outcome vectors instead of "actual probabilities" (which in practice are also just observed population proportions). If nobody else gives an answer, I'll try to do so in the next days. – Stephan Kolassa Jul 29 '21 at 21:17
  • Hi, @StephanKolassa. Do you find out something? Thank you. – igorkf Aug 10 '21 at 12:29
  • Now I'm justing using deterministic outcomes (0 or 1) to calculate. Thanks – igorkf Aug 26 '21 at 18:39

0 Answers0