12

I have a data mining assignment where I make a content-based image retrieval system. I have 20 images of 5 animals. So in total 100 images.

My system returns the 10 most relevant images to an input image. Now I need to evaluate the performance of my system with a Precision-Recall curve. However, I do not understand the concept of a Precision-Recall curve. Let's say my system returns 10 images for a gorilla image, but only 4 of them are gorillas. The other 6 images returned are other animals'. Thus,

  • precision is 4/10 = 0.4 (relevants returned) / (all returned)
  • recall is 4/20 = 0.2 (relevants returned) / (all relevants)

So I only have a point, <0.2,0.4>, not a curve. How do I have a curve (i.e., a set of points)? Should I change the number of images returned (this is fixed at 10 in my case)?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
jeff
  • 1,102
  • 3
  • 12
  • 24
  • 2
    Most models assign a probability of belonging to a class, not a class itself - or you squeeze one out of a classifier. The curve is derived by changing the probability cut-off. You'll likely get more detailed answers if you mention the classifier your using. – charles Apr 17 '14 at 21:52
  • I compute feature vectors (color, texture and shape) and obtain similarity scores for each, sum them up for a total similarity score, then sort descending. the top 10 image indices are the most relevant ones. I can obtain the class index from the image index since the images are ordered (20 gorillas, 20 giraffes etc.) I hope I made myself clear, since I don't fully understand the concepts classifier / descriptor etc. – jeff Apr 17 '14 at 21:56
  • Realized I didn't read question well. Thought you had a two class problem (gorilla/no-gorilla). With more classes way beyond me, this may be helpful: http://stats.stackexchange.com/questions/2151/how-to-plot-roc-curves-in-multiclass-classification – charles Apr 18 '14 at 02:03

1 Answers1

11

Generating a PR curve is similar to generating an ROC curve. To draw such plots you need a full ranking of the test set. To make this ranking, you need a classifier which outputs a decision value rather than a binary answer. The decision value is a measure of confidence in a prediction which we can use to rank all test instances. As an example, the decision values of logistic regression and SVM are a probability and a (signed) distance to the separating hyperplane, respectively.

If you dispose of decision values you define a set of thresholds on said decision values. These thresholds are different settings of a classifier: e.g. you can control the level of conservatism. For logistic regression, the default threshold would be $f(\mathbf{x}) = 0.5$ but you can go over the entire range of $(0, 1)$. Typically, the thresholds are chosen to be the unique decision values your model yielded for the test set.

At each choice of threshold, your model yields different predictions (e.g. different number of positive and negative predictions). As such, you get a set of tuples with different precision and recall at every threshold, e.g. a set of tuples $( T_i, P_i, R_i )$. The PR curve is drawn based on the $( P_i, R_i )$ pairs.

If I understood your comment correctly, the total similarity score you compute can be used as a decision value.

Marc Claesen
  • 17,399
  • 1
  • 49
  • 70
  • This is not clear to me, can you work through with a detailed example similar to the OP's animal image retrieval situation? – M.R. Apr 12 '18 at 01:11