1

One of the hardest tasks in using machine learning methods is choosing the appropriate hyper-parameters of the model such as regularization parameter. As far as I know, this task is performed by a trial-and-error procedure, that is we try different values and select the ones which result in the better accuracies on the validation set.

I am curious to know is there any method for learning these parameters (especially regularization and learning rate parameters)? Can we simply use the backpropagation algorithm to find the appropriate values of these parameters based on the gradient descent?

Hossein
  • 3,170
  • 1
  • 16
  • 32
  • have you considered cross-validation? – user795305 Jul 21 '17 at 03:47
  • @Ben Yes, but cross-validation is a trial-and-error procedure. I am seeking for an automatic hyper-parameter determination. – Hossein Jul 21 '17 at 07:05
  • I'm not sure what you mean by that. If you select the tuning parameter with the smallest cross-validated error, then the procedure is automatic. – user795305 Jul 21 '17 at 12:22
  • @BenYes, but then you can only choose among a limited pre-determined set of hyper-parameter values. By gradient descent, you can search over a continuous space. – Hossein Jul 21 '17 at 14:10
  • These regularization parameters hurt the in-sample (or training) performance of the method, so that the parameters necessarily need to be estimated in a different way than, say, the coefficients (or weights.) It would be helpful for me if you wrote out in full detail the method of estimation that you're proposing. – user795305 Jul 21 '17 at 19:53
  • @Ben After posting the question, I found [this related paper](https://arxiv.org/pdf/1511.06727.pdf). Take a look at it. – Hossein Jul 22 '17 at 01:40

0 Answers0