3

Despite my readings (on stack 1, 2, or in literature (Cawley, 2010; Japkowicz, 2011)), I don't find a clear procedure for tuning and evaluating a model in a classification task.

I want to perform a manual features/parameters selection (Model Tuning) and, after that, perform a model evaluation with a cross-validation (k=10) on the entire dataset.

What is the best strategy? Intuitively, I would do those steps :

  1. Divide dataset in validation_train (60%), validation_test (20%) and test (20%).
  2. Process the Model Tuning on the validation_train and validation_test.
  3. Repeat (2) until a good accuracy is reached.
  4. Finally, process a CV-10fold on the entire dataset (validation_train, validation_test and test).

Or, this strategy conducts to a too optimistic score?

Ref :

  • Cawley, G. C. & Talbot, N. L. On over-fitting in model selection and subsequent selection bias in performance evaluation The Journal of Machine Learning Research, JMLR. org, 2010, 11, 2079-2107.
  • N. Japkowicz and M. Shah, Evaluating learning algorithms: a classification perspective, Cambridge University Press, 2011
Community
  • 1
jreg
  • 33
  • 3

2 Answers2

5

I would have thought it was better to

  1. Divide the dataset into training (80% or whatever) and testing (20% or whatever)

  2. Use $k$-fold cross-validation on the training set to tune your model

  3. Repeat (2) until you have optimised your model. Your model is now tuned

  4. Use your model to predict on the testing set to get an estimate of out-of-model errors

You could consider putting your full dataset back into the model (without retuning), though you would not longer be able to make statements about out-of-model errors, and the impact of widening the inputs would usually be small.

Henry
  • 30,848
  • 1
  • 63
  • 107
  • Thank you for your answer. Yes, it seems that your procedure is more reliable than my proposition. But, here, we use the k-fold CV for tuning the model. It's not possible to use a k-fold CV for the evaluation also? It has never been clear to me on what do the CV (validation_test or test, or both?) – jreg Jun 28 '14 at 10:54
  • I see *cross*-validation as sometimes using the same data for training and sometimes using it for validation and error estimation. The price you pay for this use of the same data for training and validation is potential overfitting and so less credibility for the out-of-model error estimate, which it why it may be worth holding out a test set completely from the entire training process. – Henry Jun 28 '14 at 12:33
0

Here is an approach for tuning.

Assume you have a dataset $D = {<x_1, c(x_1)>, <x_2, c(x_2)>, <x_3, c(x_3)>, <x_4, c(x_4)>, <x_5, c(x_5)>, <x_6, c(x_6)>}$ where $x_i$'s are examples and where $c(x_i)$'s are labels.

  1. Use examples from $x_1$ to $x_5$ as training data
  2. Generate some models using examples from $x_1$ to $x_4$
    1. Use different parameter values of the learning algorithm (e.g. use different $k$ values for $k$-NN such as 5, 10, 15)
  3. Validate (i.e. test) generated models over $x_5$ ($x_5$ is validation set)
  4. Repeat steps 2 and 3 to make cross validation (e.g. Generate models using examples $x_1$, $x_2$, $x_3$ and $x_5$, and validate these models using $x_4$)
  5. Once a final model is ready, test it using the test set, $x_6$
Ricardo Cristian Ramirez
  • 235
  • 4
  • 11
  • Thank you for you answer. Your answer is similar to that of Henry. In the same way, I'm wondering if it's not possible to use a k-fold CV for the evaluation, and not for the tuning? Maybe, there is a risk to do that with my proposed procedure? – jreg Jun 28 '14 at 10:59