Grid search extrapolation

Question

Hyperparameter optimization via grid search returns a value of a chosen metric for each set of hyperparameters in the grid.

Would it make sense to fit the values of the metric (target variable) using as predictors the hyperparameters?

After fit, one could use the regression to estimate the metric for values of the hyperparameters that are not in the the grid search.

I badly want to see someone do this and read their thoughts on the pros and cons. However, one issue I see is that, when you model the metric, why not tweak hyperparameters in that model and model the grid search of that model, and then model the grid search of the model of the model, etc? It seems really easy to fall in a rabbit hole and lose sight of what you really want to model. — Dave, May 30 '20 at 13:11
@Dave, I believe that going down that rabbit hole is not an issue if there is nothing that prevent doing the first step. However, that would mean trying to save time on the grid search by adding more grid searches to the problem. The first step is meant to identify the maximum of the metric. After that, one can do a finer grid search on the original model around that maximum. — Newbie, May 30 '20 at 13:51
related: https://stats.stackexchange.com/questions/193306/optimization-when-cost-function-slow-to-evaluate/193310#193310 — Sycorax, May 30 '20 at 15:45

score 1 · Answer 1 · answered May 30 '20 at 13:15

A similar but not the same approach is Bayesian HPO, where a probabilistic model is fit over the HPs (based on their validation performance) and new prospectively good HPs are guessed. So, treating your model as a black-box with HPs as inputs and validation performance as output is not an entirely new thing. Also bear in mind that, in general, extrapolation is not the strongest asset of ML algorithms.

Grid search extrapolation

1 Answers1