4

I am really confused on why should we take IRLS as it seems that using gradient that with one derivatives at a time would solve the problem, what is the meaning of introducing Hessian matrix? Or did I understand anywhere wrong with that algorithm?

xxx222
  • 397
  • 5
  • 14
  • Your question is a little ambiguous. When you say "gradient with one derivative at a time" do you mean co-ordinate descent, gradient descent or some other algorithm? – Glen_b Jan 12 '16 at 05:47
  • @Glen_b Sorry, it's gradient descent in this case. – xxx222 Jan 12 '16 at 05:49

1 Answers1

5

(It's a little hard to guess what your understanding of the algorithm is, so judging if you misunderstood something is tricky)

If "solving the problem" was the only criterion, you could ask the same of any algorithm that converged to a local minimum.

So rather than look at why use one algorithm when another can also find an optimum, let's consider "what are some potential advantages of using one algorithm over another?"

  1. In this case, because it provides second derivative information, if the problem is sufficiently "nice" and we start sufficiently near the optimum, IRLS (along with a number of related methods) may in some cases be quite fast.

  2. Secondly, by having second-derivative information "built in", one can readily obtain information about the uncertainty in parameters estimates (estimates of the standard errors and covariances of parameter estimates).

  3. Thirdly, there's a few pieces of other useful information that may be obtained reasonably easily from the final least-squares estimate (like the hat-matrix, for one example).

[This is not to suggest we should always use IRLS; it depends on the circumstances we find ourselves in -- but sometimes it may be a sensible and convenient choice. We have to weigh up the various costs and benefits.]

Glen_b
  • 257,508
  • 32
  • 553
  • 939
  • Thank you. I have supplemented the question and could you explain it with more details? – xxx222 Jan 12 '16 at 07:19
  • No, you have changed the question quite a lot - this goes far beyond clarification to a completely new set of questions. The original question was a valid question with a valid answer. Please put the previous question back (click the "edited x hours ago" link near your avatar then click "rollback" on the version you need to revert to) and ask the new questions in its own post. If it needs this one for reference you can link to this one in the new post. – Glen_b Jan 12 '16 at 08:58