1

I am trying to implement logistic regression using the following algorithm:

  1. fit a simple linear model $y \sim Xb_0$
  2. calculate $W = \frac{e^{Xb_0}}{(1+e^{Xb_0})^2}$
  3. calculate $z = Xb_0 + y \cdot \frac{(1+e^{Xb_0})^2}{e^{Xb_0}} - (1+e^{Xb_0})$ (the square and dot product means element-wise here)
  4. calculate infomation matrix $J = X^T W X$
  5. solve for a new coefficient vector $b_1 = J^{-1}X^T W z$
  6. if difference between $b_0$ and $b_1$ not small enough, assign b1 to b0, and repeat step 2-5

This sometimes work, but sometimes $b_0$ and $b_1$ will not converge. I am suspecting the initial b value was not picked optimally. Any suggestions? I can also post the c++ code here, but that may not be very helpful.

qed
  • 2,508
  • 3
  • 21
  • 33
  • Are you trying to implement IRLS? – Memming Mar 30 '14 at 15:14
  • Yes, IRLS, exactly! – qed Mar 30 '14 at 15:17
  • 2
    it's well known that IRWLS doesn't always converges for the [logit](http://www2.sas.com/proceedings/forum2008/360-2008.pdf). The most comon cause in applications is the issue of [(almost) perfect separation](http://stats.stackexchange.com/questions/45803/logistic-regression-in-r-resulted-in-hauck-donner-phenomenon-now-what/45831#45831) and it is a statistical issue (so this forum is the correct place to ask). Do your non convergences problems happen in situations where the data is perfectly separated? – user603 Mar 30 '14 at 18:59
  • Yes! That's exactly the case! For example: y = (0, 0, 0, 1, 1, 1), x = (1, 1, 1, 2, 2, 2) – qed Mar 30 '14 at 21:22
  • 1
    But it also causes trouble in some not-so-perfect cases. – qed Mar 30 '14 at 21:23
  • Should I choose a different algorithm? – qed Mar 30 '14 at 21:24
  • @user603 Also, in certain cases, the glm function in R does converge, but my program does not. – qed Mar 30 '14 at 21:48

0 Answers0