Let's suppose that I have classification model for n classes ($n>1$). The classifier returns a probability distribution over a set of classes. But if classifier is not sure (i.e. there is no probability greater than given threshold) I would like the model to return answer "none of the above". In other words it is classifier with the following decision rule:
$class(x) = \left\{\begin{matrix} \underset{i}{arg\, max} \,p_i & {if } & \exists_i\, p_i \geq \mathfrak{p} \\ \text{none-of-the-above} & {if } & \forall_i\, p_i < \mathfrak{p} \end{matrix}\right.$
where $\mathfrak{p}$ is a probability threshold and $p_i$ is a probability that object $x$ belongs to class $i$.
And the question is: how to measure model accuracy?
One idea could be a calculation of any standard metric (for example $F_1$ score) and then multiplying it by percent of predicted classes:
$quality = F_1 \cdot {{\text{size of test set}\,-\,\text{number of "none-of-the-above" cases}}\over{\text{size of test set}}}$
Is that good idea? Or there are other approaches?
