24

I know that $$\hat{\beta_0}=\bar{y}-\hat{\beta_1}\bar{x}$$ and this is how far I got when I calculated the variance:

\begin{align*} Var(\hat{\beta_0}) &= Var(\bar{y} - \hat{\beta_1}\bar{x}) \\ &= Var((-\bar{x})\hat{\beta_1}+\bar{y}) \\ &= Var((-\bar{x})\hat{\beta_1})+Var(\bar{y}) \\ &= (-\bar{x})^2 Var(\hat{\beta_1}) + 0 \\ &= (\bar{x})^2 Var(\hat{\beta_1}) + 0 \\ &= \frac{\sigma^2 (\bar{x})^2}{\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x})^2} \end{align*}

but that's far as I got. The final formula I'm trying to calculate is

\begin{align*} Var(\hat{\beta_0}) &= \frac{\sigma^2 n^{-1}\displaystyle\sum\limits_{i=1}^n x_i^2}{\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x})^2} \end{align*}

I'm not sure how to get $$(\bar{x})^2 = \frac{1}{n}\displaystyle\sum\limits_{i=1}^n x_i^2$$ assuming my math is correct up to there.

Is this the right path?

\begin{align} (\bar{x})^2 &= \left(\frac{1}{n}\displaystyle\sum\limits_{i=1}^n x_i\right)^2 \\ &= \frac{1}{n^2} \left(\displaystyle\sum\limits_{i=1}^n x_i\right)^2 \end{align}

I'm sure it's simple, so the answer can wait for a bit if someone has a hint to push me in the right direction.

QuantIbex
  • 3,880
  • 1
  • 24
  • 42
M T
  • 601
  • 1
  • 7
  • 14
  • 2
    This is not the right path. The 4th equation doesn't hold. For example, with $x_1=−1$, $x_2=0$, and $x_3=1$, the left term is zero, whilst the right term is $2/3$. The problem comes from the step where you split the variance (3rd line of second equation). See why? – QuantIbex Jul 12 '13 at 23:55
  • Hint towards Quantlbex point: variance is not a linear function. It violates both additivity and scalar multiplication. – David Marx Jul 13 '13 at 01:53
  • @DavidMarx That step should be $$=Var((-\bar{x})\hat{\beta_1}+\bar{y})=(\bar{x})^2Var(\hat{\beta_1})+\bar{y}$$, I think, and then once I substitute in for $\hat{\beta_1}$ and $\bar{y}$ (not sure what to do for this but I'll think about it more), *that* should put me on the right path I hope. – M T Jul 13 '13 at 03:53
  • This is not correct. Think about the condition required for the variance of a sum to be equal to the sum of the variances. – QuantIbex Jul 13 '13 at 10:29
  • I thought that $\bar{y}$ is considered non-random when you condition on the $x$'s, so it can be treated as a constant, i.e. $$Var(aX+b)=a^2 Var(X) + b$$. – M T Jul 13 '13 at 14:46
  • 2
    No, $\bar{y}$ is random since $y_i = \beta_0 + \beta_1 x_i + \epsilon$, where $\epsilon$ denotes the (random) noise. But OK, my previous comment was maybe misleading. Also, ${\rm Var}(aX + b)= a^2{\rm Var}(X)$, if $a$ and $b$ denote constants. – QuantIbex Jul 13 '13 at 19:38

2 Answers2

23

This is a self-study question, so I provide hints that will hopefully help to find the solution, and I'll edit the answer based on your feedbacks/progress.

The parameter estimates that minimize the sum of squares are \begin{align} \hat{\beta}_0 &= \bar{y} - \hat{\beta}_1 \bar{x} , \\ \hat{\beta}_1 &= \frac{ \sum_{i = 1}^n(x_i - \bar{x})y_i }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } . \end{align} To get the variance of $\hat{\beta}_0$, start from its expression and substitute the expression of $\hat{\beta}_1$, and do the algebra $$ {\rm Var}(\hat{\beta}_0) = {\rm Var} (\bar{Y} - \hat{\beta}_1 \bar{x}) = \ldots $$

Edit:
We have \begin{align} {\rm Var}(\hat{\beta}_0) &= {\rm Var} (\bar{Y} - \hat{\beta}_1 \bar{x}) \\ &= {\rm Var} (\bar{Y}) + (\bar{x})^2 {\rm Var} (\hat{\beta}_1) - 2 \bar{x} {\rm Cov} (\bar{Y}, \hat{\beta}_1). \end{align} The two variance terms are $$ {\rm Var} (\bar{Y}) = {\rm Var} \left(\frac{1}{n} \sum_{i = 1}^n Y_i \right) = \frac{1}{n^2} \sum_{i = 1}^n {\rm Var} (Y_i) = \frac{\sigma^2}{n}, $$ and \begin{align} {\rm Var} (\hat{\beta}_1) &= \frac{ 1 }{ \left[\sum_{i = 1}^n(x_i - \bar{x})^2 \right]^2 } \sum_{i = 1}^n(x_i - \bar{x})^2 {\rm Var} (Y_i) \\ &= \frac{ \sigma^2 }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } , \end{align} and the covariance term is \begin{align} {\rm Cov} (\bar{Y}, \hat{\beta}_1) &= {\rm Cov} \left\{ \frac{1}{n} \sum_{i = 1}^n Y_i, \frac{ \sum_{j = 1}^n(x_j - \bar{x})Y_j }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } \right \} \\ &= \frac{1}{n} \frac{ 1 }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } {\rm Cov} \left\{ \sum_{i = 1}^n Y_i, \sum_{j = 1}^n(x_j - \bar{x})Y_j \right\} \\ &= \frac{ 1 }{ n \sum_{i = 1}^n(x_i - \bar{x})^2 } \sum_{i = 1}^n (x_j - \bar{x}) \sum_{j = 1}^n {\rm Cov}(Y_i, Y_j) \\ &= \frac{ 1 }{ n \sum_{i = 1}^n(x_i - \bar{x})^2 } \sum_{i = 1}^n (x_j - \bar{x}) \sigma^2 \\ &= 0 \end{align} since $\sum_{i = 1}^n (x_j - \bar{x})=0$.
And since $$\sum_{i = 1}^n(x_i - \bar{x})^2 = \sum_{i = 1}^n x_i^2 - 2 \bar{x} \sum_{i = 1}^n x_i + \sum_{i = 1}^n \bar{x}^2 = \sum_{i = 1}^n x_i^2 - n \bar{x}^2, $$ we have \begin{align} {\rm Var}(\hat{\beta}_0) &= \frac{\sigma^2}{n} + \frac{ \sigma^2 \bar{x}^2}{ \sum_{i = 1}^n(x_i - \bar{x})^2 } \\ &= \frac{\sigma^2 }{ n \sum_{i = 1}^n(x_i - \bar{x})^2 } \left\{ \sum_{i = 1}^n(x_i - \bar{x})^2 + n \bar{x}^2 \right\} \\ &= \frac{\sigma^2 \sum_{i = 1}^n x_i^2}{ n \sum_{i = 1}^n(x_i - \bar{x})^2 }. \end{align}

Edit 2

Why do we have ${\rm var} ( \sum_{i = 1}^n Y_i) = \sum_{i = 1}^n {\rm Var} (Y_i) $?

The assumed model is $ Y_i = \beta_0 + \beta_1 X_i + \epsilon_i$, where the $\epsilon_i$ are independant and identically distributed random variables with ${\rm E}(\epsilon_i) = 0$ and ${\rm var}(\epsilon_i) = \sigma^2$.

Once we have a sample, the $X_i$ are known, the only random terms are the $\epsilon_i$. Recalling that for a random variable $Z$ and a constant $a$, we have ${\rm var}(a+Z) = {\rm var}(Z)$. Thus, \begin{align} {\rm var} \left( \sum_{i = 1}^n Y_i \right) &= {\rm var} \left( \sum_{i = 1}^n \beta_0 + \beta_1 X_i + \epsilon_i \right)\\ &= {\rm var} \left( \sum_{i = 1}^n \epsilon_i \right) = \sum_{i = 1}^n \sum_{j = 1}^n {\rm cov} (\epsilon_i, \epsilon_j)\\ &= \sum_{i = 1}^n {\rm cov} (\epsilon_i, \epsilon_i) = \sum_{i = 1}^n {\rm var} (\epsilon_i)\\ &= \sum_{i = 1}^n {\rm var} (\beta_0 + \beta_1 X_i + \epsilon_i) = \sum_{i = 1}^n {\rm var} (Y_i).\\ \end{align} The 4th equality holds as ${\rm cov} (\epsilon_i, \epsilon_j) = 0$ for $i \neq j$ by the independence of the $\epsilon_i$.

QuantIbex
  • 3,880
  • 1
  • 24
  • 42
  • I think I got it! The book has suggested steps, and I was able to prove each step separately (I think). It's not as satisfying as just sitting down and grinding it out from this step, since I had to prove intermediate conclusions for it to help, but I think everything looks good. – M T Jul 14 '13 at 00:25
  • See edit for the development of the suggested approach. – QuantIbex Jul 14 '13 at 06:19
  • The variance of the sum equals the sum of the variances in this step: $$ {\rm Var} (\bar{Y}) = {\rm Var} \left(\frac{1}{n} \sum_{i = 1}^n Y_i \right) = \frac{1}{n^2} \sum_{i = 1}^n {\rm Var} (Y_i) $$ because since the $X_i$ are independent, this implies that the $Y_i$ are independent as well, right? – M T Jul 14 '13 at 18:40
  • Also, you can factor out a constant from the covariance in this step: $$ \frac{1}{n} \frac{ 1 }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } {\rm Cov} \left\{ \sum_{i = 1}^n Y_i, \sum_{j = 1}^n(x_j - \bar{x})Y_j \right\} $$ even though it's not in both elements because the formula for covariance is multiplicative, right? – M T Jul 14 '13 at 18:42
  • Is my reasoning correct on those two points? I think it is, but I want to make sure I don't mislearn something. Since I proved the formula for the variance of $\hat{\beta_0}$ a different way, I want to make sure mine lines up with yours. – M T Jul 15 '13 at 19:00
  • Please see edit 2 for your first point. – QuantIbex Jul 15 '13 at 20:00
  • For your second point, recall that for random variables $X$ and $Y$, and constants $a$ and $b$, we have ${\rm cov} (aX, bY) = ab\,{\rm cov} (X, Y)$. – QuantIbex Jul 15 '13 at 20:02
  • I don't know what you mean by "multiplicative formula". Anyway, I guess I answered this in my previous comment. – QuantIbex Jul 15 '13 at 20:55
  • Thanks for the clarification. This is really helpful. (I would vote you up but I don't have enough rep). Was the solution I used correct, as far as you can see, or did it work by chance? – M T Jul 15 '13 at 21:02
  • I'm planning to; I just want to make sure the other solution is actually correct too and didn't just work by chance. – M T Jul 15 '13 at 21:16
  • In the accepted answer, at the end you seem to ignore the (x-bar)^2 term multiplied by the variance of Beta1. I can't find where you account for it. You seem to treat it as merely x-bar when simplifying the equation. Could you show the missing steps accounting for this? Otherwise, the equation you conclude with does not make sense. – Gabriel Oct 20 '14 at 18:50
  • @Gabriel, thanks for spotting this typo. The term $\bar{x}^2$ was missing after the first equality for $\mbox{Var}(\hat{\beta}_0)$ at the very end of Edit 1, but the subsequent expressions seem correct. – QuantIbex Oct 21 '14 at 06:51
  • For the \begin{align} {\rm Var}(\hat{\beta}_0) &= {\rm Var} (\bar{Y} - \hat{\beta}_1 \bar{x}) \\ &= {\rm Var} (\bar{Y}) + (\bar{x})^2 {\rm Var} (\hat{\beta}_1) - 2 \bar{x} {\rm Cov} (\bar{Y}, \hat{\beta}_1). \end{align} line, do you have to work out the COV part manually to show that it is zero or are the elements independent and have zero covariance by definition? – EconStats Mar 31 '15 at 17:32
  • Why is $$\frac{1}{n} \frac{ 1 }{ \sum_{i = 1}^n(x_i - \bar{x})^2 } {\rm Cov} \left\{ \sum_{i = 1}^n Y_i, \sum_{j = 1}^n(x_j - \bar{x})Y_j \right\} \\ = \frac{ 1 }{ n \sum_{i = 1}^n(x_i - \bar{x})^2 } \sum_{i = 1}^n (x_j - \bar{x}) \sum_{j = 1}^n {\rm Cov}(Y_i, Y_j) $$ – user1603548 Sep 24 '15 at 13:10
  • @user1603548, this is simply by the [properties of covariance](https://en.wikipedia.org/wiki/Covariance#Properties). – QuantIbex Sep 25 '15 at 08:43
  • If $Var(\bar{y}) = \frac{1}{n} \sum_{i=1}^n Var(y_i)$, which then becomes $\frac{1}{n} Var(\sum_{i=1}^n \epsilon_i$), why is $Var(\bar{y}) = \sigma^2 / n$ and not $\sigma^2 / n^2$? – oort Apr 07 '16 at 04:40
  • @oort, your starting point is wrong: $\mbox{Var}(\bar{Y}) = n^{-2} \sum_{i = 1}^n \mbox{Var}(Y_i)$, provided the $Y_i$s are independent. – QuantIbex Apr 07 '16 at 11:27
  • @QuantIbex, you're absolutely right, a typo on my part, but I'm confused as to where the $n$ in the numerator comes from that cancels out the $n^{-2}$ to become $n^{-1}$ – oort Apr 07 '16 at 13:29
  • 1
    @oort, in the numerator you have the sum of $n$ terms that are identical (and equal to $\sigma^2$), so the numerator is $n \sigma^2$. – QuantIbex Apr 07 '16 at 14:40
  • Not to beat a dead horse... But then $Var(\epsilon_i) = \sigma^2$ and not $Var(\epsilon)$, where $\epsilon = \sum_{i=1}^n \epsilon_i$?In ISLR they state $Var(\epsilon) = \sigma^2$, which is why I thought that $Var(\sum_{i=1}^n \epsilon_i)$ was equal to $\sigma^2$ – oort Apr 07 '16 at 15:10
  • I think the last term $n \bar{x}$ in the second equation before Edit 2 should be $n \bar{x}^2$. – rhody Nov 24 '17 at 19:42
  • I think the last term in the equation below: "And since" should be $n \bar{x}^2$ rather than $n \bar{x}$. However, excellent post, I spent days on this until I saw your post. – rhody Nov 24 '17 at 19:46
  • I'm trying to understand this. Why Ybar treated as a random variable while Xbar is not? – confused Feb 24 '20 at 18:06
1

I got it! Well, with help. I found the part of the book that gives steps to work through when proving the $Var \left( \hat{\beta}_0 \right)$ formula (thankfully it doesn't actually work them out, otherwise I'd be tempted to not actually do the proof). I proved each separate step, and I think it worked.

I'm using the book's notation, which is: $$ SST_x = \displaystyle\sum\limits_{i=1}^n (x_i - \bar{x})^2, $$ and $u_i$ is the error term.

1) Show that $\hat{\beta}_1$ can be written as $\hat{\beta}_1 = \beta_1 + \displaystyle\sum\limits_{i=1}^n w_i u_i$ where $w_i = \frac{d_i}{SST_x}$ and $d_i = x_i - \bar{x}$.

This was easy because we know that

\begin{align} \hat{\beta}_1 &= \beta_1 + \frac{\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x}) u_i}{SST_x} \\ &= \beta_1 + \displaystyle\sum\limits_{i=1}^n \frac{d_i}{SST_x} u_i \\ &= \beta_1 + \displaystyle\sum\limits_{i=1}^n w_i u_i \end{align}

2) Use part 1, along with $\displaystyle\sum\limits_{i=1}^n w_i = 0$ to show that $\hat{\beta_1}$ and $\bar{u}$ are uncorrelated, i.e. show that $E[(\hat{\beta_1}-\beta_1) \bar{u}] = 0$.

\begin{align} E[(\hat{\beta_1}-\beta_1) \bar{u}] &= E[\bar{u}\displaystyle\sum\limits_{i=1}^n w_i u_i] \\ &=\displaystyle\sum\limits_{i=1}^n E[w_i \bar{u} u_i] \\ &=\displaystyle\sum\limits_{i=1}^n w_i E[\bar{u} u_i] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i E\left(u_i\displaystyle\sum\limits_{j=1}^n u_j\right) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[E\left(u_i u_1\right) +\cdots + E(u_i u_j) + \cdots+ E\left(u_i u_n \right)\right] \\ \end{align}

and because the $u$ are i.i.d., $E(u_i u_j) = E(u_i) E(u_j)$ when $ j \neq i$.

When $j = i$, $E(u_i u_j) = E(u_i^2)$, so we have:

\begin{align} &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[E(u_i) E(u_1) +\cdots + E(u_i^2) + \cdots + E(u_i) E(u_n)\right] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i E(u_i^2) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left[Var(u_i) + E(u_i) E(u_i)\right] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \sigma^2 \\ &= \frac{\sigma^2}{n}\displaystyle\sum\limits_{i=1}^n w_i \\ &= \frac{\sigma^2}{n \cdot SST_x}\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x}) \\ &= \frac{\sigma^2}{n \cdot SST_x} \left(0\right) &= 0 \end{align}

3) Show that $\hat{\beta_0}$ can be written as $\hat{\beta_0} = \beta_0 + \bar{u} - \bar{x}(\hat{\beta_1} - \beta_1)$. This seemed pretty easy too:

\begin{align} \hat{\beta_0} &= \bar{y} - \hat{\beta_1} \bar{x} \\ &= (\beta_0 + \beta_1 \bar{x} + \bar{u}) - \hat{\beta_1} \bar{x} \\ &= \beta_0 + \bar{u} - \bar{x}(\hat{\beta_1} - \beta_1). \end{align}

4) Use parts 2 and 3 to show that $Var(\hat{\beta_0}) = \frac{\sigma^2}{n} + \frac{\sigma^2 (\bar{x}) ^2} {SST_x}$: \begin{align} Var(\hat{\beta_0}) &= Var(\beta_0 + \bar{u} - \bar{x}(\hat{\beta_1} - \beta_1)) \\ &= Var(\bar{u}) + (-\bar{x})^2 Var(\hat{\beta_1} - \beta_1) \\ &= \frac{\sigma^2}{n} + (\bar{x})^2 Var(\hat{\beta_1}) \\ &= \frac{\sigma^2}{n} + \frac{\sigma^2 (\bar{x}) ^2} {SST_x}. \end{align}

I believe this all works because since we provided that $\bar{u}$ and $\hat{\beta_1} - \beta_1$ are uncorrelated, the covariance between them is zero, so the variance of the sum is the sum of the variance. $\beta_0$ is just a constant, so it drops out, as does $\beta_1$ later in the calculations.

5) Use algebra and the fact that $\frac{SST_x}{n} = \frac{1}{n} \displaystyle\sum\limits_{i=1}^n x_i^2 - (\bar{x})^2$:

\begin{align} Var(\hat{\beta_0}) &= \frac{\sigma^2}{n} + \frac{\sigma^2 (\bar{x}) ^2} {SST_x} \\ &= \frac{\sigma^2 SST_x}{SST_x n} + \frac{\sigma^2 (\bar{x})^2}{SST_x} \\ &= \frac{\sigma^2}{SST_x} \left( \frac{1}{n} \displaystyle\sum\limits_{i=1}^n x_i^2 - (\bar{x})^2 \right) + \frac{\sigma^2 (\bar{x})^2}{SST_x} \\ &= \frac{\sigma^2 n^{-1} \displaystyle\sum\limits_{i=1}^n x_i^2}{SST_x} \end{align}

M T
  • 601
  • 1
  • 7
  • 14
  • There might be a typo in point 1; I think ${\rm var(\hat{\beta})}$ should read $\hat{\beta}$. – QuantIbex Jul 15 '13 at 22:10
  • You might want to clarify notations, and specify what $u_i$ and ${\rm SST}_x$ are. – QuantIbex Jul 15 '13 at 22:13
  • $u_i$ is the error term and $SST_x$ is the total sum of squares for $x$ (defined in the edit). – M T Jul 15 '13 at 22:37
  • Ok, in that case, the step $\hat{\beta}_1 = \beta_1 + {\rm SST}_x^{-1} \sum_{i = 1}^n (x_i-\bar{x}) u_i$ is not obvious, albeit correct. – QuantIbex Jul 16 '13 at 05:59
  • 1
    In point 1, the term $\beta_1$ is missing in the last two lines. – QuantIbex Jul 16 '13 at 06:06
  • 1
    In point 2, you can't take $\bar{u}$ out of the expectation, it's not a constant. – QuantIbex Jul 16 '13 at 06:07
  • @QuantIbex Why isn't the sample mean considered a constant? If it's not, I'm not sure how to continue the proof (shown below). \begin{align} E[(\hat{\beta_1}-\beta_1) \bar{u}] &= E[\bar{u}\displaystyle\sum\limits_{i=1}^n w_i u_i] \\ &= \displaystyle\sum\limits_{i=1}^n E[w_i \bar{u} u_i] \\ &= \displaystyle\sum\limits_{i=1}^n E[w_i] E[\bar{u} u_i] \\ &= \bar{u} \displaystyle\sum\limits_{i=1}^n E[w_i] 0 \\ \end{align} – M T Jul 16 '13 at 17:29
  • Since $u_i$ is an error term, it's random. So, adding up $n$ (independent) error terms will also be random. Hence, their average $\bar{u}$ is not a constant. For the proof, $w_i$ are constants and can be taken out of the expectation. The only thing left is computing ${\rm E}(\bar{u}u_i)$. – QuantIbex Jul 16 '13 at 17:44
  • @QuantIbex For the proof to work, $E(\bar{u}u_i)$ should be 0, right? Since $\bar{u}$ and $u_i$ aren't independent, I can't just separate them into $E(\bar{u})E(u_i)$, but I know that \begin{align} E(\bar{u}u_i) &= Cov(\bar{u}, u_i) + E(\bar{u})E(u_i) \end{align} and since \begin{align} Cov(\bar{u}, u_i) &= E[( \bar{u} - E(\bar{u}) )(u_i - E(u_i))] \\ &= E[(\bar{u} - \bar{u})(u_i - 0)] \\ &= 0 \end{align} we know that \begin{align} E(\bar{u}u_i) &= Cov(\bar{u}, u_i) + E(\bar{u})E(u_i) \\ &= 0 + \bar{u} \cdot 0 \end{align} – M T Jul 16 '13 at 18:09
  • This doesn't work because ${\rm E}(\bar{u}) \neq \bar{u}$. Hint: ${\rm E}(\bar{u}u_i) \neq 0$. – QuantIbex Jul 16 '13 at 18:17
  • By the zero conditional mean assumption, would $E(\bar{u}) = 0$? Sorry but I'm terribly lost on this; it seems like I keep doing every step wrong and it's only working by chance. Maybe I should ask a new question because I'm *really* lost now. – M T Jul 16 '13 at 19:44
  • I'm not sure what you mean by "zero conditional mean assumption". But yes ${\rm E} (\bar{u}) = n^{-1}{\rm E}(\sum_{i = 1}^n u_i) = n^{-1} \sum_{i = 1}^n {\rm E}(u_i) = 0$ since ${\rm E}(u_i) = 0, i = 1, \ldots, n$. Feel free to ask a new question, pointing out explicitly the difference such that it doesn't appear as a duplicate question of this one. – QuantIbex Jul 16 '13 at 20:01
  • @QuantIbex The [zero conditional mean assumption](http://en.wikibooks.org/wiki/Econometric_Theory/Assumptions_of_Classical_Linear_Regression_Model#Zero_Conditional_Mean) is one of the basic assumptions of SLR. – M T Jul 16 '13 at 20:34
  • @QuantIbex I don't understand how this can work is $E(\bar{u}u_i) \neq 0$, because \begin{align} E[(\hat{\beta_1}-\beta_1) \bar{u}] &= E[\bar{u}\displaystyle\sum\limits_{i=1}^n w_i u_i] \\ &=\frac{1}{n}\displaystyle\sum\limits_{i=1}^n E[w_i \bar{u} u_i] \\ &=\frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i E[\bar{u} u_i] \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i \left(Cov(\bar{u}, u_i) + E(\bar{u})E(u_i)\right) \\ &= \frac{1}{n}\displaystyle\sum\limits_{i=1}^n w_i Cov(\bar{u}, u_i) \\ \end{align} needs to equal 0 for this part of the proof to hold. – M T Jul 16 '13 at 20:35
  • The best way is to do the computation. Do you want the answer or do you want to try? – QuantIbex Jul 16 '13 at 20:40
  • @QuantIbex I'll keep trying. I'm on the right path (so far) in my last comment, right? – M T Jul 17 '13 at 02:10
  • Yes, you're on the right path. And once you've computed ${\rm cov}(\bar{u}, u_i)$, you'll need to use the definition of $w_i$ to complete the proof of step 2. – QuantIbex Jul 17 '13 at 12:52
  • @QuantIbex I didn't see your previous comment. Why isn't step 1 obvious? Since the $x_i$ are known, can't we treat $SST_x$ as a constant? Sorry I'm so stupid at this. – M T Jul 17 '13 at 21:09
  • Yes, the $x_i$ are known, and ${\rm SST}_x$ is a constant. It's the first equality, namely $\hat{\beta}_1 = \beta_1 + \frac{\displaystyle\sum\limits_{i=1}^n (x_i - \bar{x}) u_i}{{\rm SST}_x}$ that wasn't obvious to me since my "starting point" is $\hat{\beta}_1 = \frac{ \sum_{i = 1}^n(x_i - \bar{x})y_i }{ \sum_{i = 1}^n(x_i - \bar{x})^2 }$. – QuantIbex Jul 17 '13 at 21:36
  • @QuantIbex Sorry about that, I was using a formula that I proved earlier in my book (and the book has the proof so I know it's correct). I posted a new question (http://stats.stackexchange.com/q/64654/27969) about the other part since I think I got it but I want to make sure I didn't mess it up again. – M T Jul 17 '13 at 21:38
  • You've now completed point 2 in the other question, and points 3, 4, and 5 here seem fine to me. – QuantIbex Jul 18 '13 at 17:28
  • @QuantIbex Thank you! Sadly I'm only in Chapter 2 and the proofs will just get harder. I'm learning a bit on the way though so maybe they'll get easier! – M T Jul 18 '13 at 17:41