3

In machine learning, I've learned one of the ways to optimize hyperparameters of a model is to do a grid search, which tests model for evenly spaced out values of hyperparametrs and determines which combination gives best results on validation set.

Since space represented by hyperparameters and efficiency of the model can have multiple local optimas, would it make sense to use some metaheuristic search method, like genetic algorithm?

Our gene could be a binary sequence representing hyperparameter values, and our individual's fitness function could be score of the model for hyperparameters represented by it's genetic material.

Major flaw that comes into my mind is that model has to be trained over and over again, which would take a lot of time, but is there any way to compensate for that by doing less training iterations? Would that affect our result a lot? Also, when doing grid search, model also has to be trained for every hyperparameter combination, so maybe the time difference between the two wouldn't be that big?

What do you think?

Tvrtko Zadro
  • 131
  • 1
  • 2
  • 1
    Closely related: http://stats.stackexchange.com/questions/193306/optimization-when-cost-function-slow-to-evaluate/193310#193310 – Sycorax Jan 25 '17 at 15:40
  • Using GA to tune hyperparameters isn't new. For example, Frauke Friedrichs, Christian Igelm "Evolutionary tuning of multiple SVM parameters" and https://en.wikipedia.org/wiki/Neuroevolution – Sycorax Jan 25 '17 at 15:45

1 Answers1

2

You can use genetic algorithms. Yes, it will require to rerun experiments again and again but it is also true for other hyperparameter optimization methods. You can try to use warm-starts, i.e., don't train your models from scratch but to warm-start them from some previously found solutions. The latter sometimes is used for deep neural networks when searching for networks architectures.

Genetic algorithms can potentially be slow compared to other methods. However, they are relatively easy to adjust for any search space. The first GAs for hyperparameter tuning appeared about 30 years ago. For a more recent work, see "Large-Scale Evolution of Image Classifiers" by Real et al., 2017 at https://arxiv.org/abs/1703.01041

IndieSolver
  • 154
  • 6
  • I don't understand your answer. – Michael R. Chernick Sep 11 '18 at 05:13
  • 1
    If you tell me more about your background, e.g., whether you are familiar with i) GA, ii) training ML model using warm starts, iii) hyperparameter optimization with evaluations on different levels of fidelity (note "to compensate for that by doing less training iterations" mentioned in the question) , then I could help. – IndieSolver Sep 11 '18 at 05:16