3

I was looking at the Draper & Smith's 'Applied Regression Analysis', as did the person who asked:this other question on CrossValidated

In short - the variance-covariance matrix of the residuals in regression is given by $(I - H)\sigma^2$, where $H$ is the 'Hat Matrix'. So in general we must assume the residuals are not independent.

Yet whenever I read about the assumptions of regression it says the error terms should be independent (as well as having zero expectation $E[\epsilon_i] = 0$ and equal variance $Var(\epsilon_i) = \sigma^2 \forall i$.

I am missing something, or using a loose definition - my naive, inexperienced reading of this looks contradictory. Thank you, Chris

Community
  • 1
clumb
  • 57
  • 3

1 Answers1

4

Both of these facts are true. The errors are independent but the residuals are not. These should not be confused, however. The errors are unobservable while the residuals are something of our own construction. We hope that they behave similarly so as to make inference, but that is not always the case.

To understand the dependence of the residuals consider a model with an intercept at which case it is possible to show that the residuals should sum to zero This means that they cannot be uncorrelated as they should always satisfy this condition and deviations from it are not allowed. This correlation will tend to be more pronounced for small samples, for instance consider the case of $n=3$. We then should have

$$e_1 + e_2 + e_3 = 0 \implies e_1 = - e_2 - e_3$$

and it goes without saying that this will result in non-trivial correlations.

But then is it really wise to use the dependent residuals as a proxy for the independent errors? Well, strictly speaking no but the correlation becomes less of a problem as the sample size increases. It never really goes away but you can essentially disregard it and treat the residuals as the errors.

JohnK
  • 18,298
  • 10
  • 60
  • 103