"Least Squares" and "Linear Regression", are they synonyms?

Question

What is the difference between least squares and linear regression? Is it the same thing?

I'd say that ordinary least squares is one estimation method within the broader category of [linear regression](https://en.wikipedia.org/wiki/Linear_regression). It's possible though that some author is using "least squares" and "linear regression" as if they were interchangeable. — Matthew Gunn, Feb 02 '17 at 06:55
If you're doing [ordinary least squares](https://en.wikipedia.org/wiki/Ordinary_least_squares), I'd use that term. It's less ambiguous. — Matthew Gunn, Feb 02 '17 at 07:03
See also [what is a regression model](http://stats.stackexchange.com/questions/173660/definition-and-delimitation-of-regression-model). — Richard Hardy, Feb 02 '17 at 08:22

score 27 · Answer 1 · edited Feb 02 '17 at 12:18

Linear regression assumes a linear relationship between the independent and dependent variable. It doesn't tell you how the model is fitted. Least square fitting is simply one of the possibilities. Other methods for training a linear model is in the comment.

Non-linear least squares is common (https://en.wikipedia.org/wiki/Non-linear_least_squares). For example, the popular Levenberg–Marquardt algorithm solves something like:

$$\hat\beta=\mathop{\textrm{argmin}}_\beta S(\beta)\equiv \mathop{\textrm{argmin}}_\beta\sum_{i=1}^{m}\left[ y_i-f(x_i,\beta) \right]^2$$

It is a least squares optimization but the model is not linear.

They are not the same thing.

score 15 · Answer 2 · answered Feb 02 '17 at 09:27

15

In addition to the correct answer of @Student T, I want to emphasize that least squares is a potential loss function for an optimization problem, whereas linear regression is an optimization problem.

Given a certain dataset, linear regression is used to find the best possible linear function, which is explaining the connection between the variables.

In this case the "best" possible is determined by a loss function, comparing the predicted values of a linear function with the actual values in the dataset. Least Squares is a possible loss function.

The wikipedia article of least-squares also shows pictures on the right side which show using least squares for other problems than linear regression such as:

conic-fitting
fitting quadratic function

The following gif from the wikipedia article shows several different polynomial functions fitted to a dataset using least squares. Only one of them is linear (polynom of 1). This is taken from the german wikipedia article to the topic.

answered Feb 02 '17 at 09:27

Nikolas Rieble

3,131
11
36

3

We can argue the non-linear examples in the animation are actually still linear in the parameters. – Firebug Feb 02 '17 at 12:21
2

True, yet the model relation between the target and the input variable is non linear. Would you yet call the fitting "linear regression"? I would not. – Nikolas Rieble Feb 02 '17 at 14:32
4

We should distinguish between "linear least squares" and "linear regression", as the adjective "linear" in the two are referring to different things. The former refers to a fit that is linear in the parameters, and the latter refers to fitting to a model that is a linear function of the independent variable(s). – J. M. is not a statistician Feb 02 '17 at 19:52
True, I got caught up in general linear modelling semantics. – Firebug Feb 04 '17 at 18:15
3

@J.M. Many sources maintain that "linear" in "linear" regression means "linear in the parameters" rather "linear in the IVs". The WIkipedia article on [linear regression is an example](https://en.wikipedia.org/wiki/Linear_regression) is an example. Here's [another](http://blog.minitab.com/blog/adventures-in-statistics-2/what-is-the-difference-between-linear-and-nonlinear-equations-in-regression-analysis) and [another](http://math.stackexchange.com/questions/1426836/linear-regression-model-linearity-in-parameters-variables). Many statistics texts do the same; I'd argue it's a convention. – Glen_b Feb 06 '17 at 00:04
1

@Glen, prolly a later development than the stuff I read (I'm an old hand at this); they limited "linear regression" to fitting the model $y=mx+b$. – J. M. is not a statistician Mar 18 '17 at 16:25
@Nikolas If least squares is one possible loss function for a linear regression optimization problem, wouldn't nonlinear least squares likewise represent just one possible loss function for nonlinear regression? If so, what are alternative ways are there for deriving linear and nonlinear regression models without any variant of least squares? 9 times out of 10 these regression algorithms seem to boil down to OLS (such as Levenberg-Marquardt etc. mentioned in this thread). P.S. Could this be made into a separate CV question thread, or would a comment suffice? – SQLServerSteve Sep 07 '17 at 05:08
1

@SQLServerSteve I understand least squares in the sense of squared euclidean distance returning a value of dissimilarity between two vectors (e.g. predicition vs ground truth). It seems the different understanding of least squares results from both language and convention. Whereas I learned that least squares is only a loss function, others learned that least squares is an optimization problem. To answer your question, I read the wikipedia article to nonlinear least squares and it seems to me that it is rather an optimization problem than a loss function (fitting a nonlinear curve). – Nikolas Rieble Sep 07 '17 at 08:11
1

Further, it seems to me, that your additional questions could be a valid new thread since it goes more into detail and far beyond a simple "are they the same thing". – Nikolas Rieble Sep 07 '17 at 08:13
@Nikolas Thanks for the feedback, that clears up some misconceptions on my part about the differences between linear and nonlinear least squares. I'll mull over starting a separate thread – SQLServerSteve Sep 07 '17 at 14:04

"Least Squares" and "Linear Regression", are they synonyms?

2 Answers2