4

I am trying to implement a logistic regression function in c++, and not sure what algorithm to use. So far I have heard of these:

  1. Newton-Raphson
  2. IRLS
  3. Gradient descent

Are there other algorithms available? What are their pros and cons? Is there an algorithm generally recognized as superior to the others?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
qed
  • 2,508
  • 3
  • 21
  • 33
  • 3
    Pretty much any decent optimization algorithm can work under at least some circumstances. [Fisher scoring](http://en.wikipedia.org/wiki/Scoring_algorithm#Fisher_scoring) is very widely used and is convenient. – Glen_b Mar 31 '14 at 09:17
  • 2
    And if you incorporate step-halving into the algorithms they usually work well. – Frank Harrell Mar 31 '14 at 12:29
  • 2
    In the context of logistic regression, Newton's method reduces to iteratively reweighted least squares. Working this out is an excellent exercise. – Matthew Drury Jun 05 '15 at 03:28

1 Answers1

3

Partially answered in comments:

Pretty much any decent optimization algorithm can work under at least some circumstances. Fisher scoring is very widely used and is convenient. – Glen_b

And if you incorporate step-halving into the algorithms they usually work well. – Frank Harrell

Much used is iteratively reweighted least squares (IRLS), which is a reformulation of the Newton method. See here.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467