4

For instance, in linear regression, we have that

$$ Cov(e,\hat Y) = 0 $$

That is, the residuals and fitted values are uncorrelated.

Is this always true in any sample realization of residuals and fitted values? I think the answer is yes, but I'm having some difficulty intuitively seeing why the population property carries over to every sample.

Thanks for any help.

mai
  • 696
  • 4
  • 15
  • Regarding the title question, note that $P(\hat \theta = \theta) = 0 $ given that $\theta \in (a,b)$. – user158565 Oct 17 '18 at 04:31
  • 1
    Can you elaborate on that? – mai Oct 17 '18 at 04:34
  • 1
    You can try first. Generate random number from standard normal, calculate the sample mean, to see if you can get sample mean = 0. repeat many times. When I talk with non-statistical researchers, I tell them the point estimate is 100% wrong. – user158565 Oct 17 '18 at 04:37

2 Answers2

4

Does $Cov(X,Y)=0$ imply that the sample covariance between realizations of $X$ and $Y$ is always zero?

Certainly not! In fact, if you were to generate only two realisations of the pair of variables then you would almost certainly get perfect sample correlation!

To see this, consider two data points $(x_1, y_1)$ and $(x_2, y_2)$ with $x_1 \neq x_2$ and $y_1 \neq y_2$. It can easily be shown that the sample correlation of this data has a magnitude of one! That is, they have a perfect linear relationship! (This happens because any two points on a plane can be connected by a perfect straight line.)

For instance, in linear regression, we have that $Cov(e,\hat Y) = 0$. That is, the residuals and fitted values are uncorrelated. Is this always true in any sample realization of residuals and fitted values?

This is actuall a completely different question to the title question. In the specific case of a regression model using OLS estimation, the residual vector and the fitted response vector will always be orthogonal. This is a stronger property than saying that they have zero covariance in a distributional sense. It occurs because OLS estimation is equivalent to an orthogonal projection onto the column space of the design matrix, which decomposes the response vector into a fitted vector and an orthogonal residual vector.

Ben
  • 91,027
  • 3
  • 150
  • 376
  • I generated some random data and got that the covariance between the residuals and fitted values **is** zero after fitting a standard SLR model, so it seems that the population property holds at least sometimes. Is there some criteria for when it holds and when it doesn't? – mai Oct 17 '18 at 04:30
  • 1
    In regression, the situation is *stronger* than just having zero covariance at a distributional level. In that particular situation the residual vector and fitted response vector are orthogonal from the OLS fit. This is because OLS is equivalent to a projection where the residual and fitted vectors are orthogonal parts of the response (see explanation [here](https://en.wikipedia.org/wiki/Ordinary_least_squares#Alternative_derivations)). – Ben Oct 17 '18 at 04:35
  • Is this because $Cov(e,\hat Y) = Cov((I-H)Y,HY) = \sigma^2H(I-H) = 0$? I'm having a hard time distinguishing whether this is a distributional property or why it holds for any sample. – mai Oct 17 '18 at 04:39
  • But in the case you offer as a counterexample, the residual $e$ is $0$, and thus has zero covariance with all variables. – Dilip Sarwate Oct 17 '18 at 04:41
  • What do you mean by "the residual $e$ is 0"? I see that the elements sum to zero, but not every element is zero – mai Oct 17 '18 at 04:46
  • @Dilip: The title question asks about a general $X$ and $Y$, whereas the body of the question gives a more specific example of a regression residual. I am answering the title question. – Ben Oct 17 '18 at 04:52
  • Ben, OK. See my own answer which distinguishes between the two different questions. – Dilip Sarwate Oct 17 '18 at 05:03
  • @Dilip: I've edited mine to do that too, so as to clearly distinguish the two questions. Good idea. – Ben Oct 17 '18 at 05:14
  • @mikario As Ben pointed out, with two points $X=(x_1,x_2), Y=(y_1,y_2)$, we have that $\hat{Y} = (y_1, y_2)$ and $e = Y - \hat{Y} = (0,0)$ because the two points determine a straight line which _must_ be the straight line fitted by OLS regression – Dilip Sarwate Oct 17 '18 at 06:01
2

The question in the title is different from the example provided and has a different answer.

If $\operatorname{cov}(X,Y)=0$, is the sample covariance between samples from $X$ and $Y$ also necessarily zero?

The answer to this is No, the sample covariance need not be zero. The sample covariance is a random variable with mean zero and should be expected to have small value (at least compared to $\sigma_X\sigma_Y$), but it is not necessarily exactly equal to zero.

In a linear regression, is the vector $e$ of residuals orthogonal to the vector of estimates?

The answer to this question is Yes. See this answer (or the Wikipedia entry cited in a comment by @Ben on his own answer) for a geometric viewpoint as to why this is so.

Dilip Sarwate
  • 41,202
  • 4
  • 94
  • 200