6

I`m using the Newton-Raphson method for obtaining MLE for parameters for maximizing my objective function.
At each iteration, I want to check that is the Hessian matrix negative definite or not and I see the Hessian matrix is not negative definite at some iterations. So, I want to use the Fisher scoring algorithm (https://en.wikipedia.org/wiki/Scoring_algorithm#Sketch of derivation)
In this situation, the Fisher information matrix is indefinite!! Is it possible? What are the reasons for this? Are the second-order derivatives wrong? Or are values of expectations incorrect?

user321525
  • 71
  • 3
  • 2
    Could you clarify what you mean by "Fisher information matrix"? As [usually defined,](https://en.wikipedia.org/wiki/Fisher_information#Definition) this doesn't depend (in any way) on "iterations" towards a solution: it depends only on the likelihood function at the optimal parameter. – whuber Jun 02 '21 at 16:04
  • I can refer you to this link : https://en.wikipedia.org/wiki/Scoring_algorithm. I mean expected the observed information matrix calculated at obtained parameter value in previous iteration. – user321525 Jun 02 '21 at 16:35
  • Thanks. That's a gradient descent algorithm. Refer to posts about these algorithms for the (many) problems they can have. – whuber Jun 02 '21 at 19:10
  • @whuber Thank you. – user321525 Jun 03 '21 at 05:48
  • You need to be aware of the difference between the Fisher Information and the Observed Fisher Information, see for discussion https://stats.stackexchange.com/questions/154724/why-exactly-is-the-observed-fisher-information-used and https://stats.stackexchange.com/questions/188251/expected-and-observed-fisher-information – kjetil b halvorsen Jun 03 '21 at 10:12
  • @ kjetil b halvorsen Thank you. I studied your suggestions. please tell me what do we calculate in the **"Fisher Scoring Algorithm"**? – user321525 Jun 03 '21 at 10:49

1 Answers1

4

From https://en.wikipedia.org/wiki/Observed_information the observed Fisher information matrix is just the negative Hessian of the log likelihood function. If your log likelihood function is not convex then the hessian will not be positive-definite (and thus indefinite).

This is the case if you are using the "Observed Fisher Information Matrix" from https://en.wikipedia.org/wiki/Scoring_algorithm. If you are using the canonical notion of the Fisher Information Matrix (https://en.wikipedia.org/wiki/Fisher_information#Matrix_form) then it must be positive semi-definite.

Jspang
  • 116
  • 1
  • 1
  • 3
  • Thanks a lot. At the first iteration in Newton-Raphson method, I see that the observed fisher information matrix (negative Hessian) from (https://en.wikipedia.org/wiki/Scoring_algorithm ) is positive definite (for maximizing problem) and after the first iteration it becomes indefinite!! In this situation, Is it correct that I say my log likelihood function is not concave? Does it depend on staring values? – user321525 Jun 03 '21 at 05:47
  • Yes it is correct that your log likelihood function is not concave; if it were concave then it would be everywhere negative semi-definite. A function is concave if it is everywhere negative semi-definite (and convex for everywhere positive semi-definite). This does not depend on the starting value for the algorithm. – Jspang Jun 03 '21 at 15:19
  • It's worth noting that since your function is not convex, this algorithm will (almost certainly) not find the global minimum; instead I would recommend some sort of stochastic descent algorithm if a local minimum will not suffice for your use-case. – Jspang Jun 03 '21 at 15:23
  • @ Jspang Thank you. – user321525 Jun 04 '21 at 04:56