0

I have a huge dataset and want to carry out regression, such as gradient boosting. The problem is that the dataset is huge and hyperparameter optimization is computational expensive, especially I use cross validation for that.

Is it OK to do the hyperparameter optimization on subsets of the dataset? Then can I average these hyperparameters from different subsets and use that to train my model on the whole dataset?

ysakamoto
  • 103
  • 4
  • A more principled option would be to use a global optimization method such as Bayesian Optimization to keep the number of function calls as low as possible. – Sycorax Apr 06 '16 at 01:00
  • 1
    How long a single evaluation of your function takes and how long can you spare for estimation? – usεr11852 Apr 06 '16 at 01:16
  • Related, if not a duplicate: https://stats.stackexchange.com/questions/233548/is-hyperparameter-tuning-on-sample-of-dataset-a-bad-idea/234158 – DJBunk Sep 07 '18 at 16:13

1 Answers1

3

You need to use algorithms which support multi-fidelity evaluations, i.e., which are able to exploit computationally cheap surrogates of your actual/expensive objective function. Instead of running on 100% of your data you will be able to run on say 1% and 10% to get computationally cheap approximates of the optimum of your high-fidelity objective function.

Multi-fidelity approaches can lead to a significant speedup up to several orders of magnitude if low-fidelity evaluations provide some non-random information which might be exploited for high-fidelity evaluations.

IndieSolver
  • 154
  • 6