3

For linear regression, one assumption is that the target variable Y has an underlying linear relationship with features (X1, X2, . . . , Xd), modified by some error term ε that follows a zero-mean Gaussian distribution. I do not understand from where the error term comes. If Y IS the target/true label, then how there can be error in it? Is it introduced because of noise in observation?

Or it means the relationship between Y and the features is not exactly linear, hence linearity assumption introduces some errors?

Rakib
  • 495
  • 3
  • 12
  • 1
    Viewed from the perspective of no errors, then there may not be a solution to the system of equations. – Sycorax Sep 29 '16 at 21:46
  • 3
    It goes much deeper than nonlinearity. In many circumstances the *very same* vector of regressor values is assumed to give rise to *differing*, not completely predictable, values of the response $Y$. The response therefore is characterized by a distribution. That, in a nutshell, is what regression is (linear or not). – whuber Sep 29 '16 at 21:59
  • @whuber, thanks for your comment. So is this a correct statement: the underlying true relationship is unknown. So it is approximated using a polynomial curve with some error? – Rakib Sep 29 '16 at 22:13
  • I read it from @Glen_b's profile, 'Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful -- George Box & Norman R. Draper, Empirical Model-Building and Response Surfaces'. In fact, you can use a model without noise, for example, when you generate data $y=2x$, then absolutely you can use a linear regression model without noise, because you know the model and it interpolates $y$. But the fact is that you don't know what is model in practice. – Fly_back Sep 29 '16 at 23:41
  • 2
    It is the expected value of y that is taken to be a linear combination of the x's, not y itself. – dsaxton Sep 30 '16 at 00:48

2 Answers2

2

The classic linear regression model is:

$$ y_i = \beta_0 + \beta_1 x_{i,1} + \ldots + \beta_k x_{i,k} + \epsilon_i$$

The error term captures everything else that's going on besides a linear relation ship with $x_1$ through $x_k$! An entirely equivalent way to write the linear model that may be instructive is:

$$ \epsilon_i = y_i - \left(\beta_0 + \beta_1 x_{i,1} + \ldots + \beta_k x_{i,k}\right) $$

From this, you can get a sense of where linear regression can go wrong. If $\epsilon_i$ has stuff going on such that If $\mathrm{E}\left[\epsilon_i \mid X \right] \neq 0$, then strict exogeneiety is violated and the regressors and the error term are no longer orthogonal. (Orthogonality of the regressors and the error term is what gives rise to the normal equations, to the OLS estimator $\hat{\mathbf{b}} = \left(X'X\right)^{-1} X'\mathbf{y}$.)

Think of the error term as a garbage collection term, a term that collects EVERYTHING ELSE that's going on besides a linear relationship between $y_i$ and your observed regressors $x_1, \ldots, x_k$. What could end up in the error term is limitless. Of course, what's allowed into the error term for OLS to be a consistent estimator isn't limitless :P.

Matthew Gunn
  • 20,541
  • 1
  • 47
  • 85
  • I don't think that's quite right. The values of $ϵ$ are *dependent* on the choice of regressors, and therefore $ϵ$ being orthogonal to (outside the column space of) $X$ can't "give rise to" anything. It's really the other way around: By evaluating $(X^TX)^{-1}X^TY$, the *result* is that $ϵ$ is outside the column space of $X$. – Josh Oct 16 '17 at 22:30
  • @Josh Perhap there is confusion whether you're referring to $\epsilon$ as error terms or residuals. In ordinary least squares (OLS), the *residuals* are made orthogonal to regressors in your *sample*. If *error terms* are orthogonal to your regressors in the *population*, then OLS estimates parameters of interest (to the extent that the sample approximates the population). As [this question](https://stats.stackexchange.com/questions/263324/how-can-the-regression-error-term-ever-be-correlated-with-the-explanatory-variab) shows though, error terms may not be orthogonal to regressors. – Matthew Gunn Mar 01 '18 at 20:01
  • @Josh You can go ahead and run OLS and say, "hey, all I'm doing is estimating a linear, conditional expectation function." That's generally fine. You may be trying to do more though such as recover parameters from some causal model. Then for OLS to be able to uncover those parameters, to have identification, the true underlying model must be consistent with assumptions that justify OLS. – Matthew Gunn Mar 01 '18 at 20:06
  • But doesn't that presuppose correct model specification, i.e. the functional form of the response? That is just one particular application of OLS. More generally, we don't necessarily know the "true" underlying model, so I'd rather say that the normal equations arise from using calculus to minimize the expression for MSE (perhaps I'm just mincing words). On another note, isn't it the case that whatever set of predictors are included in $X$, the expected value of the error at any point in predictor space is zero? I.e., the OLS estimator is unbiased everywhere? – Josh Mar 01 '18 at 21:50
  • I only bring up the last part because if that's so, then it doesn't make sense to say "if epsilon has stuff going on such that its expected value is nonzero..." because that's an impossibility by definition of the estimator. Wouldn't that only be the case for a biased estimator, such as that which comes from ridge regression or some other regularized regression? – Josh Mar 01 '18 at 21:51
  • @Josh If you define $\epsilon$ as the projection of the random variable $y$ onto the linear span of your regressors, then yes, you're basically right. But that's often *not* what people have in mind when running linear regressions. For example, consider [this answer of mine](https://stats.stackexchange.com/a/263331/97925). – Matthew Gunn Mar 01 '18 at 21:57
  • @Josh In that example, running a linear regression to forecast grocery store spending using only the number of buns purchased might work OK, but it wouldn't work to estimate the grocery store price of buns. – Matthew Gunn Mar 01 '18 at 22:02
  • Gotcha, but I think this actually supports my view. You point out that in the second model $b_1$ is chosen such that $b_1x_1$ is uncorrelated with $u$ (i.e., error is not preexisting), but in the same breath refer to an "actual" $u$. I think this is contradictory. Referring to "actual" error values presupposes the functional form and is only one very specific application of OLS. Unless you are working the natural sciences exclusively (and even then, only maybe...), this is a very significant assumption, and I don't see any mention of that assumption in your answer to this particular question. – Josh Mar 01 '18 at 22:10
  • Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/73920/discussion-between-matthew-gunn-and-josh). – Matthew Gunn Mar 01 '18 at 23:00
0

Both classification and (linear) regression can be viewed as a supervised learning problem. Given some data (if you want call it "training data" for classification or "observations" for regression) $(x_1, y_1), \dots, (x_n, y_n)$ we have for:

  • Classification: $x_i$ are inputs and $y_i$ are true class labels.
  • Regression: $x_i$ are data points and $y_i$ are target values (values that are usually noisy coming from measurements, you can even call them measurements or observations if the term target confuses you, I think we call it target just because it's the thing we want to get).

The task in supervised learning is to find a mapping function or rule $f$ between the $x_i$ and $y_i$ such that:

  • Classification: for a new input $x_n$, $f$ finds its corresponding class $y_n$ using $y_n = f(x_n)$.
  • Regression: for a new data point $x_n$, $f$ predicts its target value using $y_n = f(x_n) + \epsilon$ i.e. defining a mapping and taking into account the noise $\epsilon$.

The difference comes from the fact that for classification we know the true classes' labels so we affect the new input to one of the classes (from a discrete set) whereas for regression we have target values that are in the real set and since we can't define a true target value (what value will you choose from the real set values?) we define a mapping $f$ that defines more or less the tendency according to the values of our observations, and add the noise to account for the possible deviation from the mapping function.

For more insights about this refer to: https://www.youtube.com/watch?v=WpxK__SK2a0&index=2&list=PLD0F06AA0D2E8FFBA

Learn_and_Share
  • 736
  • 1
  • 9
  • 18