4

I'm interested in assessing model performance on data with an ordinal categorical dependent variable. For my use case, the ideal metric would:

  1. Not assume equal intervals between classes or that recoding to a continuous scale is appropriate
  2. Be scale independent
  3. Give preference to models that rank the outcomes accurately, with higher penalties for mis-ranking classes with a larger degree of difference (e.g., Excellent > Poor > Good is better than Excellent > Very Poor > Good)
  4. Accept continuous predictions and be indifferent to their distributions

For example, suppose we have the following test set, where "response" is 5-category ordinal response and "pred1", "pred2", and "pred3" are predictions:

id      response   pred1    pred2    pred3
 1     Excellent    1.00      150       10
 2          Good     .80       39        9
 3          Good     .85       12        5
 4          Fair     .40       11        4
 5          Poor     .39       10        3
 6     Very Poor     .20        3        2
 .             .       .        .        .
 .             .       .        .        .

For my purposes, the ideal metric would score all three predictions as equally accurate since all three perfectly rank the response.

What are my options and the benefits/drawbacks to each? Bonus points for references to R packages or functions.

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
danpelota
  • 257
  • 2
  • 8

1 Answers1

11

A good measure is Somers' Dxy rank correlation, a generalization of ROC area for ordinal or continuous Y. It is computed for ordinal proportional odds regression in the lrm function in the rms package.

Frank Harrell
  • 74,029
  • 5
  • 148
  • 322
  • This is exactly what I was looking for – I just didn't realize Somers generalized to ordinal Y. (I've used your `somers2()` function from the HMisc package, but it only accepts binary Y.) Thank you! – danpelota Apr 20 '11 at 15:33
  • Franck, did you note [my question](http://stats.stackexchange.com/questions/23238/cross-validation-and-ordinal-logistic-regression) on ordinal regression, on the outcome of various function from your `rms` package? How do I interpret a given value of `Dxy` (your answer here helped me, but can you tell more)? and a cross-validate Dxy? If you don’t have time to write a long answer, a bibliographic pointer would be welcome... – Elvis Feb 21 '12 at 21:06
  • I just put an answer on that page. Best not to create 2 pages. – Frank Harrell Feb 22 '12 at 03:03