0

In attempting to calculate the average precision of an object detection model, I am wondering about an edge case. Suppose at evaluation time that for a given category, that no detections of that category occur (no true positives, no false positives), then in a post [here] (What are correct values for precision and recall in edge cases?) there is discussion of what the precision value should be since the denominator in the equation precision = TP/(TP+FP) would be 0.

One answer says to set the value of the precision to be 1 since there are no spurious results. Alternatively the precision could be set to "NAN" or some kind of error. OK.

Then for calculating the average precision of a category, used for the eventual calculation of mean average precision, the Pascal VOC competition had two such methods.

In the first method (VOC 2007) you would average the maximum precision above a recall level for recall levels, i.e.

$\frac{1}{11}\sum_{i=1}^{10} p(r_i)$ for $r_i \in \{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 \}$

where $p(r_i) = max\{precision(r'))\}$ for $r' \geq r_i$

In the second method (VOC 2012) you would interpolate between recall levels for each possible confidence score output for a category, i.e. for $n$ confidence values $c_i$ ordered by decreased score (and so increasing recall) with associated recall values $r_i$, we would have $\sum_{i=1}^{n-1} (r_{i+1}-r_{i}) p(r_{i+1})$.

I am wondering in these two cases what will happen if there are no detections (no true or false positives). It seems to me that then there would be one recall-precision pair (0, 1) (or (0, NAN) or just no pair??) for the confidence value 0.0 percent, and so for method 1 we would get 1 or some undefined result.

I do not know what you would do for method 2 since there is only one recall value.

I thought of just assigning a 0 to the average precision in this case in my code (this is for my own implementation of mAP), but I'm not sure how to proceed. Clearly assigning 1, the best possible average precision, seems wrong since having no detections at all is a bad outcome. Assigning 0 seems appropriate, but it doesn't strictly follow the formulas provided. Any insights appreciated.

IntegrateThis
  • 93
  • 6

1 Answers1

0

To answer my own question:

The average precision from Pascal VOC 2012/computing mAP COCO-Style for no detections would be $0$.

Further the first recall-precision pair used in computing the area under the curve is always $(0,1)$, and the last value is always $(1, 0)$, where according to the formula we compute (for no detections)

$(1-0)*0 = 0$, so this works with the formula. I think my issue was only using recall/precision pairs obtained from my model results, and missing the initial and last points stated above.

IntegrateThis
  • 93
  • 6