Classification, create a grey zone

Question

I want to create a "grey zone" for a binary classifier. Grey zone means, in this zone classifier should give the result "I don't know".

I denote classes with + and -.

I have good tpr , but not so good fpr. Hence I want to do

1. Change thresholds separately for class positive and negative. For instance:

If instances has probability of belonging to class + higher than 0.9 -> +

If instances has probability of belonging to class - higher than 0.7-> -

Rest is grey zone, for instance 0.5 probability for class +, 0.5 probability for class - -> grey zone

How should I set up thresholds for two classes, would ROC threshold help?

Thanks in advance.

Answer 1

Simple method: Plot the ROC curve, tip it to the right and mark the highest point. Rotate it further and mark the highest point again. The examples corresponding to those points are at your thresholds. You can tune your results by changing how far you tilt the graph.

Rigorous method with explanation: One good way to turn an ROC curve into a threshold is with a cost function - assign a cost per false positive and a different cost per false negative, evaluate the total cost of each point on your curve and choose the point with the lowest cost. In your case need to pick two thresholds, one between + and ? and another between ? and -. Use different cost functions, giving the +? boundary a higher cost for false positives and lower cost for false negatives than the ?- boundary. And ideally you'd derive the costs from how the model is actually used - how bad is a "maybe" compared to a genuine mistake, and how bad is a "maybe" instead of a correct answer?

The two methods give the same results if the slope of the tilt is the ratio of your false negative cost to false positive cost.

Answer 2

I can think of two alternatives (there might be others too) :

1. Learn a regression instead of classification. This way you obtain a "positiveness" score, and you can decide based on its value and your thresholds (you can also optimize them). Logistic regression might be helpful for this.
2. If you already know which samples to say "I don't know", make them another class, so learn a 3-class classification.

Edit

Based on your comments:

1. I would simply try thresholds in the set [0:0.05:1] (given your "positiviness" score is between 0 and 1) and choose the one that minimizes classification/regression error. @Timothy Nodine's answer might be more helpful though.
2. Why is a third class not possible? Don't you have the same features from all samples? I also assumed you already had training samples that you want your predictor to have said "rest" (i.e. samples from the "rest" class, sorry, I still see it as a separate class :) and I believe you want your predictor to do so too). If this is the case, then it is possible, just label the data accordingly. If you don't have exemplary "rest" instances, then you can try one-class classification methods. This way you obtain two classifiers, the first one (the one-class one) decides whether it is "rest" or not, and if not, the second one classifies between positive and negative. Note that this is also 3-class classification, just in a multi-stage way.

Hope it helps,