2

I am trying to better understand “tuning the hyperparameters”. I understand how to use GridSearchCV, I found the below explanation useful:

“As we do not know whether those parameters affect each other, doing it right will require that we train a classifier for every possible combination of all parameter values. Obviously, this is too tedious for us.” (from ‘Building Machine Learning Systems with Python’).

Now, my question is model agnostic (let’s say we only have certain weights to learn from data). First we train the model (learn the weights), and then we use the GridSearchCV to tune the hyperparameters…

  • How do we know that the hyperparameters do not affect the parameters, or that the parameters do not affect the hyperparameters?

  • Let’s say I train a model to calculate the weights (with default hyperparammeters) . After than, I tune the hyperparemeters… If I re-run the model with the new tuned hyperparameters, am I assured 100% that I will get the same weights?

gunes
  • 49,700
  • 3
  • 39
  • 75
Chicago1988
  • 649
  • 4
  • 16
  • 1
    A neural network is an equation. If you add a node, you’re adding other elements to the expression. Number of nodes is both the model itself, and a hyperparameter. They are not mutually exclusive. This is why Tim’s answer is so good. – EngrStudent Jul 06 '20 at 21:43

2 Answers2

4

First we train the model (learn the weights), and then we use the GridSearchCV to tune the hyperparameters

This is not correct. For each hyperparameter configuration, model is trained again and a new set of weights is estimated. Then using those weights, we test the model on validation fold. In the end, best performing hyperparameter set is selected. Then, using it, the model is trained over all the training data. You'll probably never get the same weights/parameters for your model.

gunes
  • 49,700
  • 3
  • 39
  • 75
2

Weights define the model. They are not the only element that defines the model, but for models like neural networks, linear, or logistic regression, they are the basic building blocks. So if hyperparameter tuning didn’t affect the weights, it would not impact the model, so it’d do nothing. You want your model to change after tuning, so you expect the weights to change.

Same about the other question, hyperparameters affect parameters, and the other way around. That is the reason why you need the train set for learning the parameters, validation set for tuning hyperparameters, and test set for testing the final model. If hyperparameters were independent of weights, we wouldn’t need to bother, and could just use train set for learning both.

Tim
  • 108,699
  • 20
  • 212
  • 390