10

I'm looking for a solid reference (or references) on numerical optimization techniques aimed at statisticians, that is, it would apply these methods to some standard inferential problems (eg MAP/MLE in common models). Things like gradient descent (straight and stochastic), EM and its spinoffs/generalizations, simulated annealing, etc.

I'm hoping it would have some practical notes on implementation (so often lacking in papers). It doesn't have to be completely explicit but should at least provide a solid bibliography.

Some cursory searching turned up a couple of texts: Numerical Analysis for Statisticians by Ken Lange and Numerical Methods of Statistics by John Monahan. Reviews of each seem mixed (and sparse). Of the two a perusal of the table of contents suggests the 2nd edition of Lange's book is closest to what I'm after.

chl
  • 50,972
  • 18
  • 205
  • 364
JMS
  • 4,660
  • 1
  • 22
  • 32
  • Related thread: [Optimization textbooks for statistics and data analytics](https://stats.stackexchange.com/questions/371620/). – Richard Hardy May 31 '19 at 06:31

4 Answers4

5

Nocedal and Wrights book

http://users.eecs.northwestern.edu/~nocedal/book/

is a good reference for optimization in general, and many things in their book are of interest to a statistician. There is also a whole chapter on non-linear least squares.

NRH
  • 16,580
  • 56
  • 68
5

James Gentle's Computational Statistics (2009).

James Gentle's Matrix algebra: theory, computations, and applications in statistics‎ (2007), more so towards the end of the book, the beginning is great too but it's not exactly what you're looking for.

Christopher M. Bishop's Pattern Recognition (2006).

Hastie et al.'s The elements of statistical learning: data mining, inference, and prediction‎ (2009).

Are you looking for something as low-level as a text that will answer a question such as: "Why is it more efficient to store matrices and higher dimensional arrays as a 1-D array, and how can I index them in the usual M(0, 1, 3, ...) way?" or something like "What are some common techniques used to optimize standard algorithms such as gradient descent, EM, etc.?"?

Most texts on machine learning will provide in-depth discussions of the topic(s) you're looking for.

Phillip Cloud
  • 809
  • 7
  • 22
  • The second (what are some common techniques...). Most texts present a model and then describe how to do inference. I'm looking for sort of the inverse, where the focus is on ways to fit a model and then comparing them in applications, if that makes sense. There are a few of these sort of books for MCMC where they compare different samplers and describe where they're useful & some of the pitfalls (eg Gamerman & Lopes). – JMS Mar 21 '11 at 18:07
  • Also, thanks for the references thus far. The Hastie et al book is pretty close, actually. It's been a while since I had it off the shelf; thanks for the prompt :) – JMS Mar 21 '11 at 18:10
3

Optimization, by Kenneth Lange (Springer, 2004), reviewed in JASA by Russell Steele. It's a good textbook with Gentle's Matrix algebra for an introductory course on Matrix Calculus and Optimization, like the one by Jan de Leeuw (courses/202B).

chl
  • 50,972
  • 18
  • 205
  • 364
  • @chi That book looks fantastic! Although I do agree with the reviewer that there are some conspicuous absences (simulated annealing & the various stochastic EM flavors). Kind of odd since it's in their statistics series, but c'est la vie – JMS Apr 30 '11 at 16:16
  • Also, are you familiar with Harville's matrix algebra book? I'd be curious to know how it compares with Gentle's. I find Harville a nice reference, but very dense. Just from the TOC of Gentle's book I like the whole part 2 being devoted to "selected applications" – JMS Apr 30 '11 at 16:19
  • @JMS Nope. I only have Gentle's textbook. (Because I only make a moderate usage of mathematical textbooks in general, except this one which I found pretty handy for multivariate data analysis.) Part 2 is about application (section 9) and Part 3 about software issues. The homepage is http://mason.gmu.edu/~jgentle/books/matbk – chl Apr 30 '11 at 16:25
  • Yeah, looking at it more it seems to have more from the applied side. Harville's book is very theorem-proof, but focused on results that are important in statistics; I think they probably complement each other pretty well despite the overlapping material. – JMS Apr 30 '11 at 17:27
1

As a supplement to these, you may find Magnus, J. R., and H. Neudecker (2007). Matrix Calculus with Applications in Statistics and Econometrics, 3rd ed useful albeit heavy. It develops a full treatment of infinitesimal operations with matrices, and then applies them on a number of typical statistical tasks such as optimization, MLE and non-linear least squares. If in the end of the day you will end up figuring out backward stability of your matrix algorithms, good grasp of matrix calculus will be indispensable. I personally used the tools of matrix calculus in deriving asymptotic results in spatial statistics and multivariate parametric models.

StasK
  • 29,235
  • 2
  • 80
  • 165