0

I have a series of classification tasks at hand and as a final step, I need to build a classifier for a particularly small dataset. This dataset has 9 features and the class is binary while having only 65 instances.

This is the procedure I followed:

  1. Splitting the 64 instances into a train and test set using a 90% / 10% ratio. Therefore, the train set has 57 instances and the remaining 7 instances will construct the test set.
  2. Performing model selection using a 5-fold cross-validation on the train set.
  3. Training the final model on the train set.
  4. Testing the trained model on the test set.

The test results are satisfactory at the first glance, but my only concern is that the test set contains very few instances, namely 7! Is that theoretically or practically a problem? Is this a particular situation in which one might consider performing a rather non-standard procedure? Would using an 80% / 20% ratio solve the potential problem?

Thanks in advance.

Nima Maleki
  • 1
  • 1
  • 9/10 ratio of what to what? 5-fold cross validation is on both train and test set? If not, how is that cross validation; where is the "cross." Training the final model on the train set? I do not understand what is in which sets. – Carl Mar 24 '17 at 23:53
  • Let me clarify. From the original dataset, I pick 90% of the instances for the train set, and the remaining 10% for the test set. So, my train set has ~57 instances and the train set has ~7 instances. Then I need to perform model selection. Because the datasets are already small, I don't pick a static validation set. Instead, I perform my model selection and parameter tuning using cross-validation on the train set (only). When everything is done, I test the model on the test set. – Nima Maleki Mar 25 '17 at 10:35
  • I pick 90% of the instances for the train set, and the remaining 10% for the test set. So, my train set has ?~?57 instances and the $test$? set has ?~?7 instances. Then I need to perform model selection. Because the datasets are already small, I don't pick a ?static? validation set. Instead, I perform my model selection and parameter tuning using ? *cross-validation on the train set (only)* ** – Carl Mar 25 '17 at 16:20

1 Answers1

1

With such a small test set, the variance in the estimate of your classification accuracy is going to be very high.

Once you've trained a model $f$ on your training set, you then want to estimate its accuracy, i.e. $$p = \Pr_{(x, y) \sim \mathcal P}(f(x) = y).$$ You usually evaluate that with an empirical estimate $$\hat p = \frac{1}{n} \sum_{i=1}^n \mathbb 1(f(x_i) = y_i),$$ where $\{(x_i, y_i)\}_{i=1}^n$ is your test set. If your model has true accuracy $p$, then note that $n \hat p \sim \mathrm{Binomial}(n, p)$, which tells us that $\mathbb E[ \hat p ] = p$ (good!) and $\mathrm Var[\hat p] = \frac{p (1-p)}{n}$. So if your true $p$ is, say, $\frac34$, then with $n = 7$, $\mathrm Var[\hat p] = \frac{3}{112} \approx .027$, which doesn't seem so terrible until you realize that corresponds to a standard deviation of $0.16$. That's a pretty significant standard deviation for your test accuracy!

It might help to do nested cross-validation, which will take longer but give you somewhat more reliable test accuracies.

Community
  • 1
Danica
  • 21,852
  • 1
  • 59
  • 115
  • 2
    Thanks a lot for your answer. This was actually my concern. I posted this a few days ago and since then I have come to the idea that if using different train/test fragmentations, I can see if the error or accuracy is stable all over. But the systematic way of doing that would actually be nested cross-validation. Thanks again for elaborating! – Nima Maleki Mar 31 '17 at 12:57