0

I'm using a data set of 32 face persons and a svm-rbf to classify and a random group of 70% for train and 30% for test.

The problem is that my results are heavily dependent of the random group used for train the svm. Depending of random group sometimes I have 100% true positive rate whit 10% false positive rate, and whit another group I have 60% tpr and 30% fpr. On average the best performance is 70% tpr and 20% fpr.

What can I say about my data?

What can I do to improve my results?

(pca, lda and pca+lda don't improve the results) (I'm trying clustering now)

Felipe Aplaplac
  • 1
  • 2
  • 1
    Get more samples. If you are trying to build a classifier using only 32 samples you are aiming for something that is most likely not reproducible. What is your goal with the study? Do you want to explore data, or build an actual classifier for a practical purpose? – demodw Feb 18 '16 at 16:06
  • We are classifying drunk people, so the samples are low. Is just for study. – Felipe Aplaplac Feb 18 '16 at 16:36
  • You seem to have a small data set. For larger data sets, your performance metric could vary a lot with the break-up of the data, if the data for your classifier is skewed (e.g. very few drunks compared to non-drunks). See, for example, http://florianhartl.com/thoughts-on-machine-learning-dealing-with-skewed-classes.html – Salmonstrikes Feb 19 '16 at 00:54

1 Answers1

0

To address one of your problems, which is performance estimation, I suggest you perform the hold-out procedure multiple times (with different splits) to get an idea of the variance of the performance estimation. You can use the average over all iterations as your final performance estimate. Even better would be to use cross-validation and to repeat it multiple times (5 folds may be reasonable in your case).

It is not clearly stated in the question, but I assume you do not actually perform any model selection (that is, you train only a single model on your train data). In case you are actually also performing model selection, you have to keep an additional validation set which is used to select the best hyper-parameter configuration. In this case I would suggest you to use a nested cross-validation protocol for performance estimation and a cross-validation for model selection (see this post for a description of the procedure). Again, I would suggest to repeat the procedure multiple times.

Community
  • 1
George
  • 419
  • 3
  • 8