7

Page 24 of Introduction to Statistical Learning states:

...linear regression is a relatively inflexible approach, because it can only generate linear functions such as the lines shown in Figure 2.1 or the plane shown in Figure 2.3.

Later on, linear regression is defined more formally as assuming $f$ takes on form

$$ f(X) = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \ldots + \beta_p X_p $$

and "solving" for the $\beta_i$ using techniques like the "ordinary least squares" method (here I'm using the notation from the book).

Question: If this is really how linear regression is defined, why do we say that "linear regression [can]...only generate linear functions such as [lines and planes]"?

For example, if our model for $\hat{Y}$ is as follows:

$$ \hat{Y} = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \ldots + \beta_p X_p $$

then any (or all) of the $X_j$ could be themselves highly non-linear or otherwise pathological functions that take on many bizarre shapes that look nothing like lines or planes. That being the case, a linear combination of those pathological random variables could end up looking like something that fails to resemble a line or a plane. Doesn't this show the quote from above to be false?

George
  • 587
  • 2
  • 8
  • 4
    Good question. A very closely related one at http://stats.stackexchange.com/questions/148638 concerns what "linear" in "linear model" might possibly mean. – whuber Apr 22 '16 at 16:19
  • You may be also happy with [this post](http://stats.stackexchange.com/a/92087/67822), regarding the comment by @whuber on my initial (now erased) answer. – Antoni Parellada Apr 22 '16 at 17:16
  • @whuber: just to be clear, that post indicates that there are several notions of linearity, and that -- at the very least -- the quote above is wrong (in the sense that it is indeed possible for a model to be considered linear but in no way to resemble a line or a plane). Is my understanding correct here? – George Apr 22 '16 at 17:39
  • 1
    I believe your understanding is correct, but I wouldn't go so far as to characterize the quote as "wrong": if you look at it the right way, such a linear regression indeed only fits lines, planes, and other linear subspaces. I think the fundamental point is that you--the modeler--begin by establishing a mapping from a space of possible regressor values into a space of regressors (such as mapping numbers $x$ to vectors $(1,x,x^2)$). Although that mapping may be nonlinear, the model *in terms of the image vectors* is perfectly linear. – whuber Apr 22 '16 at 18:14
  • The data can all be non-linear but when you run linear regression or perform least squares you can be sure that you'll get a line. Your model for Y hat leaves off the residuals. All of that non linearity in the predictors you discussed will just be part of the residuals around the fitted line after OLS. The line may not describe much of the variance, but it's still a line. – conv3d Apr 23 '16 at 09:10

1 Answers1

1

Unfortunatelly my rating is too low to add a comment, that's why I have to answer.

Just imagine that there is some feature $Z$. $X_1 = \sin Z, \ X_2 = \cos Z$, and $f(X) = \beta_0 + \beta_1 X_1 + \beta_2 X_2 = \beta_0 + \beta_1 \sin Z + \beta_2 \cos Z$.

Will $f(X)$ be linear if the coordinate system is $(Y, Z)$? And what about $(Y, X_1, X_2)$?

Maassa
  • 21
  • 5