0

Given a classifier $f$, $N$ possible classes, and an input $x$, $f$ produces a class from $[1,..., N]$ and its matching confidence $[0,...,100]$. Then I run $f$ on a large set of examples $X$, and I have the empirical results. The goal is to select N thresholds, $[t_1,...,t_N]$ such that $f$ on $X$ maximizes recall under constrained precision.

Details: For each class C and $x$, a specific example of class C:

  • the True Positives $TP$ are the amount of examples $x$, such that $f$($x$) gives class $c$ with confidence >= $t_C$.
  • the False Positives $FP$ are the amount in which $f$($x$) gives a class different from $c$ with confidence >= $T_C$.
  • The False Negatives $FN$ are the amount in which $f$($x$) gives any class with confidence < $T_C$.

The recall is the sum over all classes of $\frac{TP}{TP+FN}$, and precision is the sum over all classes of $\frac{TP}{TP+FP}$.

In other words, we select many thresholds to optimize a single global recall score under constrained global precision, i.e., maximize the number of found examples under the constraint of 90% precision across all seen examples.

I looked at convex optimization, but Recall/Precision are non-convex; I'm not sure how to proceed from here.

TIA

Danny
  • 53
  • 4
  • 1
    What do you mean by "$f$ produces **a** class"? Does $f$ produce class membership probabilities for all classes, or only for one (which one?)? Also, be aware that precision and recall are highly dubious measures, for all the same reasons as accuracy: [Why is accuracy not the best measure for assessing classification models?](https://stats.stackexchange.com/q/312780/1352) [Is accuracy an improper scoring rule in a binary classification setting?](https://stats.stackexchange.com/q/359909/1352) [Classification probability threshold](https://stats.stackexchange.com/q/312119/1352) – Stephan Kolassa Oct 10 '21 at 20:29
  • Thanks, in this context, we only look at the top-1 class and its matching score. – Danny Oct 11 '21 at 08:09

0 Answers0