2

Following scenario:

  • I trained an object detector (Viola Jones) on images of 8 subjects and evaluated on three other subjects (the test set) and achieved recall and precision values of roughly 82% and 80% respectively.
  • I trained Viola Jones on images of 14 subjects (including the above mentioned 8) and evaluated on the same test set and achieved recall and precision of 80% and 78%.

This was of great surprise to me, since I assumed the test set error should decrease with the increase of training set size. What could be possible reasons for this behaviour?

I can only think of:

  1. Chance. But what exactly does that tell us about the method (Viola Jones) used?
  2. The additional data is noisy. Possible, but unlikely...

PS: I also evaluated all subjects in scenario 1 and scenario 2 in a cross-validation manner, where I always left one subject out as test. Here, the performance drop in scenario 2 was even slightly more significant.

user1809923
  • 233
  • 1
  • 10

2 Answers2

1

Recall and precision (and accuracy) depend on your observations. Your observations are random variables. Therefore your KPIs are also random variables. They have an expectation and a variance.

Plugging a particular set of observations into your model fitting algorithm and into your evaluation will lead to a particular realization of recall and precision.

The rule of thumb that more training data will lead to lower error refers to expectation. Even if the expectation of one random variable (e.g., precision from a large training set) is smaller than the expectation of a different one (e.g., precision from a large training set), a given realization from the first one may still be larger than one from the second one. For instance, if you draw one N(0,1) and one N(1,1) variable, the first one will still quite often come out larger than the second one.

And: eight, or even fourteen, samples are very little.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • To clarify: subjects does not refer to samples, but individuals. What does it tell me about my classifier and/or dataset, when my KPIs have a high variance? – user1809923 Nov 29 '17 at 09:34
  • If your KPI has a high variance, then your results will vary a lot if you apply your model to different new datasets. That is, it is quite probable that model $A$ outperforms model $B$ on some *specific* new dataset, even though *in expectation* $B$ outperforms $A$. – Stephan Kolassa Nov 29 '17 at 10:36
0

when you refer 14 subject. you mean 14 pictures? if so, I think 8 or 14 pictures is a very small data set. so the result should be pretty random.

scott huang
  • 111
  • 3