1

From taking some online linear regression courses (Stanford Machine Learning and John Hopkins Linear Regression) it seems that there are at least three ways of finding a the coefficients of a linear regression:

  1. Minimizing the r^2 error using gradient descent (or other optimization function)
  2. Solving for the coefficients using matrices
  3. Calculating the slope directly using Cov(x,y)/Var(x) and the intercept using mean(y) - slope*mean(x)

I can understand the reasons for choosing between gradient descent and solving using matrices (i.e. the cost of inverting a matrix or for singular matrices) - but I don't get why you wouldn't always just use the simple closed form solution. Its computationally a lot less expensive, and easier to understand.

What are the reasons for choosing to use optimization or solving over the simple closed form solution? Is it that the closed form solution doesn't work for multiple regressors? or does the closed form generalize somehow to the matrix solution?

Brian Flynn
  • 111
  • 2
  • 2
    +1 because while that Stanford ML course is excellent, I find that it creates this unnecessary confusion to people. It is extremely rare in general LR application to see optimisation approaches. Almost always it will be a matrix decomposition (usually QR). See the excellent answers provided in the threads [here](https://stats.stackexchange.com/questions/154485) and [here](https://stats.stackexchange.com/questions/278755). (And yes, you are correct one can rewrite $Cov(x,y)/Var(x)$ in matrix algebra terms, consider what $X^TX$ is when $X$ is zero-meaned... – usεr11852 Jul 16 '17 at 09:40
  • Perfect, thanks for pointing me to answers @usεr11852 - very useful and explains it clearly. – Brian Flynn Jul 16 '17 at 10:36
  • The use of general optimization procedures to find least squares coefficients is *always* just to illustrate the technique. Optimization procedures come into their own when applied to other problems that do not have closed-form solutions: in other words, to objective functions that aren't perfect quadratic forms. – whuber Jul 16 '17 at 17:36

0 Answers0