0

I had a query about Data set splitting.

Say, I have a data set and I split them into 3 different sets - Training Set, Validation Set and Test Set. I will use the Training Set and Validation set to go over different algorithms and choose the best performing one (Based on validation Set accuracy and all)

Now, I am convinced that a particular algo (model) with certain parameters does well (Since I have validated them on my validation set).

I finally take that algo (model) selected and run the Test set. Here are the questions -

  1. Is this the accuracy (Test Set Accuracy) I need to report?
  2. What if it performs really bad on test set? What do I do next?
  3. If I re-work the whole process wouldn't it be like using the Test set for choosing an Algo (model)?
  4. Ideally after Test set is applied I shouldn't be going back to whiteboard for new algo selection/ tuning?

Appreciate all the time.

Uno
  • 121
  • 2

1 Answers1

0

  1. Yes, you should report test set accuracy because this should be representative of how well the model generalises.

  2. If it performs poorly on the test set then either you have overfit to both your training and validation data or something very funny is going on with how you've split the data.. You should probably have picked this up on the validation set, or earlier. Using the test set should be the last thing you do for model building, used solely to see how well your model generalises to unseen data. You should consider bootstrapping or cross-validating on the training data so you don't over fit to your validation set.

  3. Yes. That's why it should be the last thing you do.

  4. No. I think this is appropriate when you are looking at the validation results, but not testing.

I think it might be worth reading this question: What is the difference between test set and validation set?

MachineEpsilon
  • 2,686
  • 1
  • 17
  • 29
  • Does that mean we will also have to understand the Test data with respect to Training and Validation data better, so as to make sure there isn't much variation per se. Are there any specific techniques to do that? – Uno Jan 25 '18 at 09:58
  • What process is being followed say if the Test set result is bad? And how often does that happen in real use cases? – Uno Jan 25 '18 at 10:01
  • These are pretty general questions that is hard to answer without more details of your problem. Andrew Ng suggest a slightly different approach here outlined here http://www.computervisionblog.com/2016/12/nuts-and-bolts-of-building-deep.html?m=1 . Look at the section on **The 5-step method of building better systems** . – MachineEpsilon Jan 25 '18 at 10:24