Identity in Simple Linear Model

Question

I'm working on this identity $$\sum_{i-1}^n (y_i - \hat {\beta_0} - \hat {\beta_1}x_i)^2 = \sum_{i=1}^n y_i^2 - \hat {\beta_0}\sum_{i=1}^n y_i - \hat {\beta_1} \sum_{i=1}^n x_iy_i$$

I have these relationships to work with:

$$\hat {\beta_1} = \frac { n\sum_{i_1}^n x_iy_i - \left ( \sum_{i=1}^n x_i \right ) \left ( \sum_{i=1}^n y_i \right )}{ n \left ( \sum_{i=1}^n x_i^2 \right ) -\left (\sum_{i=1}^n x_i \right )^2 }$$

$$ \hat {\beta_0}= \overline {y} - \hat {\beta_1} \overline {x}$$

A little manipulation also shows

$$\hat {\beta_1}= \frac { \sum_{i=1}^n ( x_i - \overline {x}) y_i}{ \left ( \sum_{i=1}^n x_i^2 \right ) -n \overline {x}^2 }$$

My strategy has to substitute $\hat {\beta_1}$ out of the equation, but I just keep getting $\overline {y}$ in too many terms and not any terms with $y_i$. The formula for $\hat {\beta_1}$ is too complicated to consider working with. What am I missing?

Answer 1

You are trying to reprove the Pythagorean Theorem. Understanding the connection provides a powerful intuition for understanding ordinary least squares regression and (incidentally) makes short work of the proof.

---

Let $y$ represent the vector $(y_1, y_2, \ldots, y_n)$, $\mathbf{1}$ the $n$-vector $(1,1,\ldots,1)$, and $x$ the vector $(x_1,x_2, \ldots, x_n)$. Denote by $\hat{y}$ the vector $\hat\beta_0\mathbf{1} + \hat\beta_1 x.$ In this notation the identity (after combining the last two sums) is

$$ ||y-\hat y||^2 = \sum_{i-1}^n (y_i - \hat {\beta_0} - \hat {\beta_1}x_i)^2 = \sum_{i=1}^n y_i^2 - \sum_{i=1}^n\left(\hat {\beta_0} - \hat {\beta_1} x_i\right)y_i = ||y||^2 - \hat y \cdot y.$$

Because $\hat\beta_0$ and $\hat\beta_1$ are formulas for the least-squares coefficients, by definition $y-\hat{y}$ minimizes the squared distance from $y$ to the line generated by $\hat{y}$. Because $y$ and $\hat{y}$ span a space of at most two dimensions, understanding their relationship is a matter of planar Euclidean geometry which is faithfully illustrated with a simple diagram:

The formulae for $\hat\beta_0$ and $\hat\beta_1$ are usually derived by demonstrating the geometrically obvious fact that $y-\hat y$ must be perpendicular to $\hat y$. In terms of vector operations, this means their dot product is zero:

$$\hat y \cdot (y - \hat y) = 0.$$

Expanding this dot product shows it is equivalent to the key relationship

$$||\hat y||^2 = \hat y \cdot \hat y = \hat y \cdot y.$$

Consider the Pythagorean Theorem. In this right triangle it asserts that the square of one leg equals the square of the hypotenuse minus the square of the other leg:

$$||y-\hat y||^2 = ||y||^2 - ||\hat y||^2.$$

The key relationship provides another expression for the last term, yielding

$$||y-\hat y||^2 = ||y||^2 - \hat y \cdot y,$$

which is the desired identity, QED.

Answer 2

OK, I will do some parts and leave the rest for you to do it yourself. I dropped the index of summations for simplicity. Start from expanding the L.H.S to have $$L.H.S=\sum y_i^2+\sum(\hat{\beta_0}+\hat{\beta_1}x_i)^2-2\hat{\beta_0}\sum y_i-2\hat{\beta_1}\sum (y_ix_i)$$ which is $$L.H.S=\Big[\sum y_i^2-\hat{\beta_0}\sum y_i-\hat{\beta_1}\sum (y_ix_i)\Big]+\sum(\hat{\beta_0}+\hat{\beta_1}x_i)^2-\hat{\beta_0}\sum y_i-\hat{\beta_1}.\sum (y_ix_i).$$ Now what we have inside the bracket is actually the R.H.S. So you need to show that the rest is zero i.e. $$\sum(\hat{\beta_0}+\hat{\beta_1}x_i)^2-\hat{\beta_0}\sum y_i-\hat{\beta_1}\sum (y_ix_i)=0.$$ Now to show this, I will give you some hints. You need to do them correctly and step by step.

1. Replace $\hat{\beta_0}$ by $\bar{y}-\hat{\beta_1}\bar{x}$ to re-write it all based on $\hat{\beta_1}$.
2. Expand the terms and simplify (some terms will be canceled out).
3. Now use two facts:
   (1): $\hat{\beta_1}=\dfrac{S_{xy}}{S_{xx}},$ where $S_{xy}=\sum(x_i-\bar{x})(y_i-\bar{y})$ and $S_{xx}=\sum(x_i-\bar{x})^2$ and
   (2): $S_{xy}=\sum (x_iy_i)-\dfrac{\sum x_i .\sum y_i}{n}$
   to write everything in terms of $S_{xy}$ and $S_{xx}$.
4. Simplify to show that it is zero.

Answer 3

Alternatively, one can show that $$\sum(\hat{\beta_0}+\hat{\beta_1}x_i)^2-\hat{\beta_0}\sum y_i-\hat{\beta_1}\sum y_ix_i=0.$$ by regrouping it to $$\hat{\beta_0} \sum(y_i-\hat{\beta_0}-\hat{\beta_1}x_i)+\hat{\beta_1}\sum (y_i-\hat{\beta_0}-\hat{\beta_1}x_i)x_i=0.$$ and invoking the minimisation argument $$0=\frac{\partial}{\partial{\hat{\beta_0}}}\sum(y_i-\hat{\beta_0}-\hat{\beta_1}x_i)^2=2\sum(y_i-\hat{\beta_0}-\hat{\beta_1}x_i)$$ $$0=\frac{\partial}{\partial{\hat{\beta_1}}}\sum(y_i-\hat{\beta_0}-\hat{\beta_1}x_i)^2=2\sum(y_i-\hat{\beta_0}-\hat{\beta_1}x_i)x_i$$