4

I tried to reproduce the experiments described in this paper and wanted to compare the output of my system with the one described in the article. I am looking for a statistical comparison of the systems (I'm not really confident when it comes to hypothesis testing).

To be more precise the dataset consists of images of certain categories of rooms (bathroom, living room, etc.) acquired in 6 different houses. In the experiment, we use 5 houses to train a classifiers and the 6th to test it (the author call it leave-one-out but strictly speaking I'm not sure it's exactly that). The results is the correct classification rate for each room in each "fold".

Do you have any idea?

Mathieu Dubois
  • 181
  • 6

1 Answers1

1

This is really what is called the holdout method. Some samples are used to construct or train the classifier and the rest for testing. Leave one out fits the classifier on all but one case and tests on that one case and then repeats this process for each case. Leave-one-out produces a nearly unbiased estimate for the error rate. If the holdout sample is randomly selected the holdout estimate is unbiased. The error rate estimates are simply the number classified as belonging to group A out of the cases that are actually from A. You would do the saem for group B. In your case you have as many classes as there are rooms in the homes. Of course 1-error rate estimate is the estimate of the correct classification probability for the room. So that is how you calculate the estimate. If you want confidence intervals for the estimated probabilities you could use the bootstrap.

Michael R. Chernick
  • 39,640
  • 28
  • 74
  • 143