Constructing Scores for Machine Learned Ranking on a Labeled Dataset with Classifiers and Regressors

Question

I have a dataset containing students test grades on different classes and personal information, as well as a label from the set {'abandoned college', 'graduated', 'master', 'phd'} which describes how far that student went on his academic life.

My goal was to generate a continuous score that is directly proportional to how far one student is likely to reach on his academic life. (e.g. on a 0-10 scale, a 1 student is unlikely to go to masters and a 9 student is very likely to go reach PhD).

(This idea was to compare this new metric with the GPA)

My initial approach was to solve this as a classification problem and use the classifier built in probabilities estimators as a score. But because it's a multiclass label, I end up with multiple probabilities. Also, those probabilities frequently are very sparse values.

Because the labels have an ordering, I thought of using a Regression instead, and just scale the predicted value. But, even though that works numerically I'm not sure it makes sense theoretically.

Is my goal even reachable, or is my problem ill defined? I've seen similar questions mentioning Ordered Regression, but I think it would not be enough (just better than ordinary classification). Should I combine Classification and Regression or just pick one?

UPDATE As suggested by a friend, I tried ensemble stacking. In a first layer I use two classifiers and in the second layer, a Regressor. This made no significant difference in the output (although it could be bad execution from my part) and I think that it makes a little bit more sense theoreticaly - the regressor is learning how to balance each classifier's opinion.

UPDATE2 It looks like L2R would be my best bet, but there's still a problem of missing data. Because each student has taken different classes, the only way to compare them directly would be to generate standard features commonly available. So the problem also imposes a problem of classification with missing features, which is hard on its own merit.

Answer 1

As you have already mentioned, your response variable is ordinal. So it makes sense to try a classifier that can handle this type of variables. For example, Frank and Hall [1) proposed a simple to implement ordinal classifier. Their approach can be adapted to almost any type of classifier however, if the number of classes is low you might not get an accuracy improvement compared with a traditional classifier as stated by the authors and also empirically tested in [2].

Also, be aware that since you are dealing with an ordinal classification problem, standard performance metrics may not be informative of the classification error since they are designed for unordered classes. These metrics treat all errors as equal. For example, confusing abandoned college with phd is more severe than confusing master with phd. For this reason, you may want to use additional performance metrics such as mean squared error, mean absolute error, linear correlation, accuracy within n [3].

You may also be interested in Learning to Rank algorithms.

1- E. Frank and M. Hall, “A simple approach to ordinal classification,” in 636 Proc. 12th Eur. Conf. Mach. Learning, 2001, vol. 2167, pp. 145–156. 637 [40] L. Gaudette and N. Japkowicz, “Evaluation methods for ordinal classifi- 638 cation,” Adv. Artif. Intell., vol. 5549, pp. 207–210, 2009.

2- Automatic Stress Detection in Working Environments from Smartphones' Accelerometer Data: A First Step. Garcia-Ceja, E., Venet Osmani, Oscar Mayora. Biomedical and Health Informatics, IEEE Journal of, vol. 20, no. 4, pp. 1053-1060, July 2016.

3- L. Gaudette and N. Japkowicz, “Evaluation methods for ordinal classification” Adv. Artif. Intell., vol. 5549, pp. 207–210, 2009.