3

Lets say I did the following steps:

  1. Used some separate development set to select some features.
  2. Decided a priori to use only one learning algorithm (SVM) with only default parameter values.
  3. Trained a single model on a training set.
  4. Tested this model on the test set.

Is it OK that I didn't use a validation set, given that I had only one model a priori?

Is this acceptable in a scientific work?

Have in mind that the purpose of my work was only to show that the feature selection was good by showing that even some standard learning algorithm with its default parameter values can learn these features and get good accuracy. I don't claim that I've found the best learning method for my problem (my work is on IR and only uses ML, it's not about ML).

zvisofer
  • 161
  • 2
  • 9

3 Answers3

3

Thinking of training/test/validation as involving different subsets of the data is not necessarily a good idea. First of it it takes enormous samples to be able to get precise accuracy estimates when data splitting. More precise estimates of likely future performance of predictive models can be had by using rigorous Efron-Gong "optimism" bootstrapping using the whole sample to develop the model and the whole sample to get a nearly unbiased estimate of future performance for observations from the same stream.

Note that even if you pre-specify a single model, validation may be needed if there are many parameters in the model.

Regarding acceptance in scientific work, see the end of Chapter 9 of Biostatistics for Biomedical Research at http://biostat.mc.vanderbilt.edu/ClinStat .

Frank Harrell
  • 74,029
  • 5
  • 148
  • 322
0

No, I don't think you need a validation set in your case. ML is (broadly speaking) carried in these three phases:

  1. Training phase: Here, we try to learn the best approach for prediction.

  2. Validation/Test phase: We calculate the performance for our model(s) using validation/test set.

  3. Application phase: We apply our (final) chosen model to get the predictions.

Now, the Validation/Test phase comprises of two sub-parts:

  1. Validation: We look at our different models, and using the validation set choose the best performing one.

  2. Testing: We test the accuracy of our selected model on test set.

As you have already decided on the model beforehand, validation set is not needed.

B.Shankar
  • 131
  • 1
  • 4
  • I always thought that test set and validation set are the same thing, like: you train a model, then *test* it on a *validation set* to verify whether it performs well. Not having a distinct test/validation set you can test it on a training set or choose 80% of features to be training set and the remaining to be test/validation set. – corey979 May 24 '15 at 11:59
  • 2
    Testing Vs Validation is an important concept in ML. You might find this http://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set helpful. – B.Shankar May 24 '15 at 12:05
  • 1
    Ah, so validation is for choosing a model (if multiple models are applied) and test is for testing the final, chosen among others, model. I recently had only one model so only tested it; that's why I didn't differentiate between the two. – corey979 May 24 '15 at 13:15
  • Right. That's the fundamental difference. – B.Shankar May 24 '15 at 13:16
  • and why not just skipping the validation set at all even when you have already chosen a model? – Ben Oct 08 '19 at 06:52
0

If there are no parameters for tuning, or tuning doesn't require trading off something at the cost of something else (in other words either increasing or decreasing a parameter is known to enhance the accuracy of the model), I don't see any reason for having a validation set.

Thoran
  • 241
  • 2
  • 5