0

I have given the function for a regression $Y_i=\beta_0+\beta_1x_1+\epsilon_i$ and I'm asked to show a proof that $RSS=n\sigma_Y^2(1-r^2)$ given $r=\frac{Cov(X,Y)}{\sigma_X\sigma_Y}$, but I don't see how this should be done.

I'm currently toying with $RSS = \sum_{i=1}^n(y_i-(\beta_0+\beta_1x_i))$ and trying to boil this down to $RSS=n\sigma_Y^2(1-r^2)$ but im not seeing how.

Tobias Christensen
  • 3
  • 1
  • 1
    You can find an algebraic derivation in several threads here on CV, I believe. I posted a geometric demonstration at https://stats.stackexchange.com/a/71303/919. – whuber Mar 31 '21 at 21:01
  • While I am in utter awe of the detailed nature of the answer to the question, I do not see how I, from the geometric answer, should derive the proof for this problem – Tobias Christensen Mar 31 '21 at 21:16
  • The geometric answer *is* a perfectly rigorous proof. You could also use it as a guide to writing an algebraic answer. For instance, you could start with the equation given for the regression line near the end -- that's probably an equation you know. – whuber Mar 31 '21 at 21:18
  • I do not even recognize where in the answer you explain a proof or derive the squared residuals. I see that you use the $1-\rho^2$ as a scalar for the conditional $Y|X$ and $\epsilon=Y-\rho X$, but how this is derived or equates to my original question I just do not see, sorry. – Tobias Christensen Mar 31 '21 at 21:44

1 Answers1

0

You have $\text{RSS} = \sum e_i^2 = \sum (Y_i -\beta_0-\beta_1X_i)^2$.

Using the linearity of the expected value we have that

$$\sum \frac{e_i}{n} = E[e]= 0\\=E[(Y -\beta_0-\beta_1X)] = E[Y] -\beta_0-\beta_1E[X]$$

So $\beta_0 = E[Y] -\beta_1E[X]$

$\beta_1$ can be obtained from this:

$$\frac{\partial \text{RSS}}{\partial \beta_1} = -2\sum ( (Y_i -\beta_0-\beta_1X_i) X_i)$$

Setting this to zero

$$\sum ( (Y_i -\beta_0-\beta_1X_i) X_i)=0\\ =\sum ( Y_iX_i -\beta_0X_i-\beta_1X_i^2)\\ =\sum Y_iX_i - \beta_0\sum X_i-\beta_1\sum X_i^2$$

Substituting $\beta_0$ and dividing by $n$:

$$0= \sum \frac{Y_iX_i}{n} - (E[Y] -\beta_1 E[X])\sum \frac{X_i}{n}-\beta_1\sum \frac{X_i^2}{n}\\ =E[YX] - (E[Y] -\beta_1 E[X])E[X]-\beta_1E[X^2]\\ =E[YX] - E[Y]E[X] +\beta_1 (E[X]^2-E[X^2]) $$

Recognizing that $\text{Cov}(Y,X) = E[XY]-E[X]E[Y]$ and $\text{Var}(X) = E[X^2]-E[X]^2$

$$\beta_1 = \frac{\text{Cov}(Y,X)}{\text{Var}(X)}$$

Plugging back into the definition of the $\text{RSS}$

$$\text{RSS} = \sum (Y_i -E[Y] +\beta_1E[X]-\beta_1X_i)^2\\ =\sum (Y_i -E[Y] -\beta_1(X_i - E[X]))^2\\ =\sum ((Y_i -E[Y])^2 + \beta_1^2(X_i - E[X])^2 - 2\beta_1(X_i - E[X])(Y_i -E[Y]) )$$

Breaking this down:

\begin{cases} \sum (Y_i -E[Y])^2 = n\text{Var}(Y) \\\sum \beta_1^2(X_i - E[X])^2 = n\beta_1^2 \text{Var}(X) = n \left(\frac{\text{Cov}(Y,X)}{\text{Var}(X)}\right)^2 \text{Var}(X) = n \frac{\text{Cov}^2(Y,X)}{\text{Var}(X)} \\2\sum \beta_1(X_i - E[X])(Y_i -E[Y]) = 2n\frac{\text{Cov}(Y,X)}{\text{Var}(X)}\text{Cov}(Y,X) = 2n\frac{\text{Cov}^2(Y,X)}{\text{Var}(X)} \end{cases}

So

$$\text{RSS} =n\text{Var}(Y) + n \frac{\text{Cov}^2(Y,X)}{\text{Var}(X)} - 2n\frac{\text{Cov}^2(Y,X)}{\text{Var}(X)}\\ =n\text{Var}(Y) - n\frac{\text{Cov}^2(Y,X)}{\text{Var}(X)}$$

Can you do the rest?

Firebug
  • 15,262
  • 5
  • 60
  • 127
  • That is the perfect solution, thank you so much! – Tobias Christensen Mar 31 '21 at 23:45
  • @TobiasChristensen If my answer suffices, you can consider accepting it (the green check mark). This marks the question as answered, which is a good thing for the site :) – Firebug Apr 01 '21 at 00:15
  • I have a few questions if you do not mind. (1) Why do we set $\partial RSS / \partial \beta_1$ = 0? the best reason I can think of, is that we are minimizing the RSS as the OLS would always tend towards this spot, but I'm not sure that's the reason. (2) As it turns out, i do not see how we go from $n \sigma_Y - n \frac{Cov(Y,X)^2}{\sigma_X^2}$ to a version where it's no longer subtracted but instead multiplied and with a $\sigma_Y^2$ in the denuminator. Can you show or give a hint how to get there? – Tobias Christensen Apr 01 '21 at 02:21
  • @TobiasChristensen (1) Your intuition is poiting you to the actual reason. $\text{RSS}$ is a convex function of the parameters $\beta_0, \beta_1$, so as we seek a solution that minimizes $\text{RSS}$, it's partial derivative will be zero at that point. (2) Hint: $$\text{Cor}(X,Y) = \frac{\text{Cov}(X,Y)}{\sqrt{\text{Var}(X)\text{Var}(Y)}}$$ – Firebug Apr 01 '21 at 11:42
  • I just do not see how to use the correlation formula from $n\frac{Cov(Y,X)^2}{Var(X)}$. as I do not have the std dev of y in the function at all – Tobias Christensen Apr 01 '21 at 12:13
  • @TobiasChristensen $$\text{RSS} =n\left(\text{Var}(Y) -\frac{\text{Cov}^2(Y,X)}{\text{Var}(X)}\right) =n\text{Var}(Y)\left(1 -\frac{\text{Cov}^2(Y,X)}{\text{Var}(X)\text{Var}(Y)}\right)$$ – Firebug Apr 01 '21 at 12:26
  • Ahh first setting $n$ outside of a parenthesis and then $\sigma_Y^2$ outside of the parenthesis, adding a $\sigma_Y^2$ in the denominator was ingenious! This makes so much sense, but I'm too dense to get anywhere near it myself! Thank you so much! – Tobias Christensen Apr 01 '21 at 21:46
  • @TobiasChristensen No problem, glad to be of help, and everyone has to start somewhere :) – Firebug Apr 01 '21 at 21:53