Many already known optimization techniques rely on past data (Bayesian optimization for instance) and perform really well for a bunch of hyperparameters. Is there, however, a good tuner/tuning method that does well with thousands of hyperparameters at the same time?
Asked
Active
Viewed 38 times
2
-
+1 There are a number of suggestions for hyper-parameter tuning here https://stats.stackexchange.com/questions/193306/optimization-when-cost-function-slow-to-evaluate/193310#193310 but I don't think any of them are suited to thousands of dimensions. The reason it's challenging to answer your question directly is that non-convex optimization is hard, non-convex optimization of black-box functions is harder, and it's always more challenging to do a task in thousands of dimensions compared to, say, 3. – Sycorax Jan 01 '21 at 17:24
-
A paper that seems relevant is is [Optimizing Millions of Hyperparameters by Implicit Differentiation](https://arxiv.org/abs/1911.02590) (2019). – Jon Nordby Jan 04 '21 at 09:53