Optimal Feature Engeneering creation: best optimization method?

Question

basically I would like to solve this problem:

(1) say I have N features that I want to transform with a generic f(x, theta) where theta is a continuous bounded variable (2) I know that each variable has got an optimal different value of theta (3) these features will be fed into a simple regression model, of which my objective is to maximize R2

Now I can set this up as an optimization problem where I minimize -R2 as a bounded optimization problem. the question is: which is the "optimal" algorithm to use in this kind of cases? optimal = relatively good at finding "minima" with relatively few iteractions

I am familiar with the methods described here (https://docs.scipy.org/doc/scipy/reference/optimize.html) but perhaps there's something different to go about this kind of problem. maybe what I am looking for is something different altogether. I am especially troubled by getting stuck in local minima and by running a lot of iterations (which would be costly from a computational perspective.

p.s.: for the purpose of this, please know I have no "true"/"prior" thing that I can tether to, just some educated guesses as to the starting points.

Answer 1

This problem would seem to fit relatively nicely into the framework of using cross-validation (CV) + (Bayesian) hyperparameter optimization. Basically, by using an appropriate from of CV you get a relatively realistic assessment of different hyperparameter choices and then you can fit a surrogate model (such as a Bayesian Gaussian process model) for how hyperparameters relate to cross-validated performance (performance = some metric you care about evaluated on the out-of-fold predictions). The idea is that such a model can both estimate where the best values might be, but also estimate where there's the greates uncertainty with a large potential upside about how good hyperparameter values are. That then let's you trade-off exploration vs. exploitation.

There are a lot of implementations for this kind of surrogate model such as those in the hyperoptor optuna Python packages, or tune::tune_bayes() in R. For most of these packages you can specify a couple of hyperparameter combinations that should be tried first before starting the automated search, which can be a way of ensuring a good human guess is not overlooked.

Some types of local minima are easier to deal with (i.e. they are just a little dip and trying something not that far away can you get out of the local minimum), while others can be very, very hard to deal with (i.e. there's an easy to find local minimum and a very far away global minimum that's perhaps quite sharp). I would expect the standard hyperparameter optimization frameworks to deal decently with the more begnin types, but not with the really nasty ones. To help at least with some of those, simulated annealing could be something to look into, but it also cannot solve every case and may be somewhat less efficient.