3

Had a question on a quiz I chose a (I and III) but the teacher said d.(all the above)

Which of the following statements about residuals are true?

I. The means of the residuals is always zero

II. The regression line for a residual plot is a horizontal line.

III. A definite pattern in the residual plot is an indication that a nonlinear model will show a better fit to the data than the LSRL

I don't think II is correct. If you have three residuals (-1,0,1) then you would have a line threw all three points.

Could someone help me understand what my mistake is? The regression line minimizes the distance between the points and the line, thus with my example (-1,0,1) that fits on a straight line with r =1.

Community
  • 1
MaoYiyi
  • 529
  • 3
  • 7
  • 15
  • did "d" mean "all of the above?" – Peter Ellis Jan 11 '13 at 04:29
  • 4
    Can you contrive a case where, after the ordinary least squares fit, you have three equally spaced residuals (-1,0,1) in that order? If you think through why this is impossible it might help. – Peter Ellis Jan 11 '13 at 04:32
  • @PeterEllis Do you know of a good online resources that will walk me threw the mathematics and not just state it without any proof? – MaoYiyi Jan 11 '13 at 04:37
  • You appear to be confused; II is true for both simple and multiple regression if this 'residual plot' is a plot of residuals against any predictor, or against the fitted values. (Residuals could be plotted against other things for which it may not be true.) – Glen_b Jan 11 '13 at 09:44
  • 2
    "Residual plot" typically refers to a plot of the residuals on the y axis and an independent variable on the x-axis. If you then draw a regression line of the residual on the IV, it has to be flat because the linear relationship between the IV and the DV has already been accounted for in the original regression. – Peter Flom Jan 11 '13 at 11:54
  • 1
    Eight such residual plots appear in the middle of my post at http://stats.stackexchange.com/questions/46185/question-on-how-to-normalize-regression-coefficient/46508#46508. The figure is a scatterplot matrix and the relevant residual plots have regression lines drawn in red: all are horizontal, as @Peter Flom argues. (A ninth red line, at the lower right, would not be possible to obtain in ordinary circumstances, because it relies on information that is not usually available.) – whuber Jan 11 '13 at 15:29
  • @whuber, thank you for such a clear and detailed post. – MaoYiyi Jan 12 '13 at 09:53

1 Answers1

2

Suppose your data is $\{(x_i,y_i)\}$ with non constant $x_i$, and you choose an arbitrary line $y=\beta_0 + \beta_1 x $ with residuals $r_i = y_i - \beta_0 - \beta_1 x_i$ associated with it.

Now suppose that the least squares regression line of $\{r_i\}$ on $\{x_i\}$ gives $r=\hat\beta_2 + \hat\beta_3 x $. Since this is a least squares regression line, if at least one of $\hat\beta_2$ or $\hat\beta_3$ is non-zero then you have $$\sum_i(r_i - \hat\beta_2 - \hat\beta_3 x_i)^2 \lt \sum_i r_i^2$$ or written another way $$\sum_i(y_i - (\beta_0 + \hat\beta_2) - (\beta_1 + \hat\beta_3)x_i)^2 \lt \sum_i(y_i - \beta_0 - \beta_1 x_i)^2.$$

So your original arbitrary line can only minimise the sum of squares of the residuals if both $\hat\beta_2$ and $\hat\beta_3$ are zero. $\hat\beta_3=0$ means a horizontal regression line of the residuals.

Henry
  • 30,848
  • 1
  • 63
  • 107