3

I have got confused between these three: Training score, Validation score, Testing score

  1. Is validation score and testing score the same thing?
  2. How is training score calculated?

For e.g., cross_validate returns test_score, train_score.

How and on what, the test_score and train_score is measured?

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html

Community
  • 1
vikramsjn
  • 131
  • 1
  • 1
  • 3
  • If your question is not fully answered by answers at the indicated duplicate, please edit to clearly focus the question on the parts that are not dealt with – Glen_b Nov 04 '17 at 11:52

1 Answers1

1

When building predictive models, it's common practice to split your data into three sets which you have correctly identified as training, validation and test.

The purpose of these splits are simple:

You train your model using the training set. In the case of a supervised classification problem, you would feed in your data with its classification for the learning algorithm to learn.

The test set is used to evaluate the performance of your model. Essentially, you do not supply your model with labels but instead what your model to predict.

The validation set is usually sampled from the training set and is used in conjunction with the training set during training used to fine tune some of your hyper parameters according to some metric.

I.e. if training a neural network, a hyper parameter you may wish to tune is the weight decay term based on the SSE metric.

Essentially the validation scores and testing scores are calculated based on the predictive probability (assuming a classification model). The reason we don't just use the test set for validation is because we don't want to fit to the sample of "foreign data". We instead want models to generalise well to all data.

This is by no means an extensive answer and you should research this further.

What is the difference between test set and validation set?

https://www.quora.com/Should-I-split-my-data-to-train-test-split-or-train-validation-test-subset

Sam
  • 175
  • 1
  • 11