27

Precision is defined as:

p = true positives / (true positives + false positives)

Is it correct that, as true positives and false positives approach 0, the precision approaches 1?

Same question for recall:

r = true positives / (true positives + false negatives)

I am currently implementing a statistical test where I need to calculate these values, and sometimes it happens that the denominator is 0, and I am wondering which value to return for this case.

P.S.: Excuse the inappropriate tag, I wanted to use recall, precision and limit, but I cannot create new Tags yet.

Björn Pollex
  • 1,223
  • 2
  • 15
  • 18
  • 1
    I don't think we need limit tag. –  Aug 17 '10 at 11:08
  • 1
    Presumably you're attempting to quantify performance of some diagnostic procedure; is there any reason you're not using a proper signal detection theory metric like d', A', or area under the ROC curve? – Mike Lawrence Aug 17 '10 at 11:19
  • 4
    @Mike, precision and recall are common evaluation metrics in, e.g., information retrieval where ROC, or in particular specificity is awkward to use because you already expect a high number of false positives. – user979 Aug 17 '10 at 18:02

4 Answers4

21

Given a confusion matrix:

            predicted
            (+)   (-)
            ---------
       (+) | TP | FN |
actual      ---------
       (-) | FP | TN |
            ---------

we know that:

Precision = TP / (TP + FP)
Recall = TP / (TP + FN)

Lets consider the cases where the denominator is zero:

  • TP+FN=0 : means that there were no positive cases in the input data
  • TP+FP=0 : means that all instances were predicted as negative
Amro
  • 468
  • 6
  • 15
  • 13
    Extending your answer: If TP=0 (as in both cases), recall is 1, since the method has discovered all of none true positives; precision is 0 if there is any FP and 1 otherwise. –  Aug 17 '10 at 11:44
  • Doesn't this answer contradict the fact that 0/0=undefined? – eod Dec 09 '21 at 22:49
  • @eod not really, because we don't really compute precision/recall in those case, it's just a special case we test `if ({special_case}) return {some_value} else return {computation}`. Besides these measures are meaningless when assessing models if for example you don't have positive cases in the data, since they can't distinguish between a good model from a trivial one that always predicts negative. – Amro Dec 10 '21 at 13:35
  • Do you have a mathematical proof for this statement? I still don't understand the answer. What I understand is: if we have real=(0,0,0,0), and predicted=(0,0,0,0), then TP=0, TN=4, FP=0, and FN=0. Therefore Recall=0/(0+0)=0/0(=undefined), and Precision=0/(0+0)=0/0(=undefined). I tried to calculate the same case in the R's "caret" package using "confusionMatrix" and it returns "Error in confusionMatrix.default(data = dat$pred, reference = dat$real, : there must be at least 2 factors levels in the data". I didn't check the code, but I hope the error is due to the fact that 0/0=undefined. – eod Dec 10 '21 at 15:54
  • `there must be at least 2 factors levels in the data` the error is telling you that the function expects both positive and negative classes to be represented in the data (two-class problem), again this is checked before doing any computation: https://github.com/cran/caret/blob/master/R/confusionMatrix.R#L177 – Amro Dec 13 '21 at 14:00
12

Answer is Yes. The undefined edge cases occur when true positives (TP) are 0 since this is in the denominator of both P & R. In this case,

  • Recall = 1 when FN=0, since 100% of the TP were discovered
  • Precision = 1 when FP=0, since no there were no spurious results

This is a reformulation of @mbq's comment.

John Lehmann
  • 221
  • 2
  • 4
4

I am familiar with different terminology. What you call precision I would positive predictive value (PPV). And what you call recall I would call sensitivity (Sens). :

http://en.wikipedia.org/wiki/Receiver_operating_characteristic

In the case of sensitivity (recall), if the denominator is zero (as Amro points out), there are NO positive cases, so the classification is meaningless. (That does not stop either TP or FN being zero, which would result in a limiting sensitivity of 1 or 0. These points are respectively at the top right and bottom left hand corners of the ROC curve - TPR = 1 and TPR = 0.)

The limit of PPV is meaningful though. It is possible for the test cut-off to be set so high (or low) so that all cases are predicted as negative. This is at the origin of the ROC curve. The limiting value of the PPV just before the cutoff reaches the origin can be estimated by considering the final segment of the ROC curve just before the origin. (This may be better to model as ROC curves are notoriously noisy.)

For example if there are 100 actual positives and 100 actual negatives and the final segnemt of the ROC curve approaches from TPR = 0.08, FPR = 0.02, then the limiting PPV would be PPR ~ 0.08*100/(0.08*100 + 0.02*100) = 8/10 = 0.8 i.e 80% probability of being a true positive.

In practice each sample is represented by a segment on the ROC curve - horizontal for an actual negative and vertical for an actual positive. One could estimate the limiting PPV by the very last segment before the origin, but that would give an estimated limiting PPV of 1, 0 or 0.5, depending on whether the last sample was a true positive, a false positive (actual negative) or made of an equal TP and FP. A modelling approach would be better, perhaps assuming the data are binormal - a common assumption, eg: http://mdm.sagepub.com/content/8/3/197.short

Thylacoleo
  • 4,829
  • 5
  • 24
  • 32
2

That would depend on what you mean by "approach 0". If false positives and false negatives both approach zero at a faster rate than true positives, then yes to both questions. But otherwise, not necessarily.

Rob Hyndman
  • 51,928
  • 23
  • 126
  • 178
  • 2
    I really do not know the rate. To be honest all I know is that my program crashed with a division-by-zero and that I need to handle that case somehow. – Björn Pollex Aug 17 '10 at 09:54