49

In simple linear regression, we have $y = \beta_0 + \beta_1 x + u$, where $u \sim iid\;\mathcal N(0,\sigma^2)$. I derived the estimator: $$ \hat{\beta_1} = \frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2}\ , $$ where $\bar{x}$ and $\bar{y}$ are the sample means of $x$ and $y$.

Now I want to find the variance of $\hat\beta_1$. I derived something like the following: $$ \text{Var}(\hat{\beta_1}) = \frac{\sigma^2(1 - \frac{1}{n})}{\sum_i (x_i - \bar{x})^2}\ . $$

The derivation is as follow:

\begin{align} &\text{Var}(\hat{\beta_1})\\ & = \text{Var} \left(\frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} \right) \\ & = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2} \text{Var}\left( \sum_i (x_i - \bar{x})\left(\beta_0 + \beta_1x_i + u_i - \frac{1}{n}\sum_j(\beta_0 + \beta_1x_j + u_j) \right)\right)\\ & = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2} \text{Var}\left( \beta_1 \sum_i (x_i - \bar{x})^2 + \sum_i(x_i - \bar{x}) \left(u_i - \sum_j \frac{u_j}{n}\right) \right)\\ & = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2}\text{Var}\left( \sum_i(x_i - \bar{x})\left(u_i - \sum_j \frac{u_j}{n}\right)\right)\\ & = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2}\;\times \\ &\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;E\left[\left( \sum_i(x_i - \bar{x})(u_i - \sum_j \frac{u_j}{n}) - \underbrace{E\left[\sum_i(x_i - \bar{x})(u_i - \sum_j \frac{u_j}{n})\right] }_{=0}\right)^2\right]\\ & = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2} E\left[\left( \sum_i(x_i - \bar{x})(u_i - \sum_j \frac{u_j}{n})\right)^2 \right] \\ & = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2} E\left[\sum_i(x_i - \bar{x})^2(u_i - \sum_j \frac{u_j}{n})^2 \right]\;\;\;\;\text{ , since } u_i \text{ 's are iid} \\ & = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2}\sum_i(x_i - \bar{x})^2E\left(u_i - \sum_j \frac{u_j}{n}\right)^2\\ & = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2}\sum_i(x_i - \bar{x})^2 \left(E(u_i^2) - 2 \times E \left(u_i \times (\sum_j \frac{u_j}{n})\right) + E\left(\sum_j \frac{u_j}{n}\right)^2\right)\\ & = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2}\sum_i(x_i - \bar{x})^2 \left(\sigma^2 - \frac{2}{n}\sigma^2 + \frac{\sigma^2}{n}\right)\\ & = \frac{\sigma^2}{\sum_i (x_i - \bar{x})^2}\left(1 - \frac{1}{n}\right) \end{align}

Did I do something wrong here?

I know if I do everything in matrix notation, I would get ${\rm Var}(\hat{\beta_1}) = \frac{\sigma^2}{\sum_i (x_i - \bar{x})^2}$. But I am trying to derive the answer without using the matrix notation just to make sure I understand the concepts.

TooTone
  • 3,631
  • 24
  • 33
mynameisJEFF
  • 1,583
  • 4
  • 24
  • 29
  • 2
    Yes, your formula from matrix notation is correct. Looking at the formula in question, $1-\frac1{n}\,=\,\frac{n-1}{n}$ so it rather looks as if you might used a sample standard deviation somewhere instead of a population standard deviation? Without seeing the derivation it's hard to say any more. – TooTone Mar 03 '14 at 00:51
  • General answers have also been posted in the duplicate thread at http://stats.stackexchange.com/questions/91750. – whuber Mar 29 '14 at 20:50
  • Related: https://stats.stackexchange.com/q/64195/119261. – StubbornAtom Jan 29 '20 at 19:49
  • what is ui and uj in the second line? – confused Feb 24 '20 at 17:38

3 Answers3

44

At the start of your derivation you multiply out the brackets $\sum_i (x_i - \bar{x})(y_i - \bar{y})$, in the process expanding both $y_i$ and $\bar{y}$. The former depends on the sum variable $i$, whereas the latter doesn't. If you leave $\bar{y}$ as is, the derivation is a lot simpler, because \begin{align} \sum_i (x_i - \bar{x})\bar{y} &= \bar{y}\sum_i (x_i - \bar{x})\\ &= \bar{y}\left(\left(\sum_i x_i\right) - n\bar{x}\right)\\ &= \bar{y}\left(n\bar{x} - n\bar{x}\right)\\ &= 0 \end{align}

Hence

\begin{align} \sum_i (x_i - \bar{x})(y_i - \bar{y}) &= \sum_i (x_i - \bar{x})y_i - \sum_i (x_i - \bar{x})\bar{y}\\ &= \sum_i (x_i - \bar{x})y_i\\ &= \sum_i (x_i - \bar{x})(\beta_0 + \beta_1x_i + u_i )\\ \end{align}

and

\begin{align} \text{Var}(\hat{\beta_1}) & = \text{Var} \left(\frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} \right) \\ &= \text{Var} \left(\frac{\sum_i (x_i - \bar{x})(\beta_0 + \beta_1x_i + u_i )}{\sum_i (x_i - \bar{x})^2} \right), \;\;\;\text{substituting in the above} \\ &= \text{Var} \left(\frac{\sum_i (x_i - \bar{x})u_i}{\sum_i (x_i - \bar{x})^2} \right), \;\;\;\text{noting only $u_i$ is a random variable} \\ &= \frac{\sum_i (x_i - \bar{x})^2\text{Var}(u_i)}{\left(\sum_i (x_i - \bar{x})^2\right)^2} , \;\;\;\text{independence of } u_i \text{ and, Var}(kX)=k^2\text{Var}(X) \\ &= \frac{\sigma^2}{\sum_i (x_i - \bar{x})^2} \\ \end{align}

which is the result you want.

As a side note, I spent a long time trying to find an error in your derivation. In the end I decided that discretion was the better part of valour and it was best to try the simpler approach. However for the record I wasn't sure that this step was justified $$\begin{align} & =. \frac{1}{(\sum_i (x_i - \bar{x})^2)^2} E\left[\left( \sum_i(x_i - \bar{x})(u_i - \sum_j \frac{u_j}{n})\right)^2 \right] \\ & = \frac{1}{(\sum_i (x_i - \bar{x})^2)^2} E\left[\sum_i(x_i - \bar{x})^2(u_i - \sum_j \frac{u_j}{n})^2 \right]\;\;\;\;\text{ , since } u_i \text{ 's are iid} \\ \end{align}$$ because it misses out the cross terms due to $\sum_j \frac{u_j}{n}$.

TooTone
  • 3,631
  • 24
  • 33
  • I noticed that I could use the simpler approach long ago, but I was determined to dig deep and come up with the same answer using different approaches, in order to ensure that I understand the concepts. I realise that first $\sum_j \hat{u_j} = 0$ from normal equations (FOC from least square method), so $\bar{\hat{u}} = \frac{\sum_i u_i}{n}=0$, plus $\bar{\hat{u}} = \bar{y} - \bar{\hat{y}} = 0 $, so $\bar{y} = \bar{\hat{y}}$. So there won't be the term $\sum_j \frac{u_j}{n}$ in the first place. – mynameisJEFF Mar 07 '14 at 13:56
  • ok, in your question the emphasis was on avoiding matrix notation. – TooTone Mar 07 '14 at 14:01
  • Yes, because I was able to solve it using matrix notation. And notice from my last comment, I did not use any linear algebra. Thanks for your great answer anyway^.^ – mynameisJEFF Mar 07 '14 at 14:04
  • sorry are we talking at cross-purposes here? I didn't use any matrix notation in my answer either, and I thought that was what you were asking in your question. – TooTone Mar 07 '14 at 14:06
  • sorry for misunderstanding haha... – mynameisJEFF Mar 07 '14 at 14:21
  • I was a little confused about this step: "noting only ui is a random variable". Can you point to a resource that explains a little about why you're able to ignore the other two terms? – Sam Ritchie Nov 20 '15 at 20:55
  • whoops, never mind. You're pulling out a constant from the variance. Missed the squared terms on the next line. – Sam Ritchie Nov 20 '15 at 20:58
  • I am also confused why we can think of x_i and x^{bar} as constants. Aren't they Random Variables? Isnt an estimator just a transformation(or function) of n i.i.d random variables(corresponds to n experiments/obervations)? – aha Dec 11 '15 at 04:35
  • when we ask questions about mean or variance, don't we need to go back to the distribution instead of using the specific realization from the experiment? Sorry I am a bit new to this, and really appreciate if someone can point to some resources regarding this. thanks – aha Dec 11 '15 at 04:38
  • @aha the $x$s are observations and are usually taken to be fixed. The random variable is the model for observations. In this case the random variable is a normal distribution of fixed variance whose mean is a linear function of an $x$. Re your second point, this could be worth a question on this site (so long as it doesn't exist already, of course). – TooTone Dec 11 '15 at 21:44
  • @aha-even I had the same confusion. The way I convinced myself was to think of $x_{i}s$ as realizations of a random variable $x_{i}$. On the other hand each $u_{i}$ is itself a random variable and the set of $u_{i}s$ is IID – wabbit Apr 05 '16 at 09:21
  • Is $\sum_i (x_i - \bar{x})^2 Var(u_i) = \sum_i (x_i - \bar{x})^2 \sum_i Var(u_i)$, therefore after $\sum_i (x_i - \bar{x})^2$ is cancelled out, we are left with $\sum_i Var(u_i) = \sigma^2$ ? – oort Apr 07 '16 at 02:50
  • I know this is ancient but $x_i$ is not an RV in this context because we're implicitly conditioning on x. In other words, "given these observations, what's the variance of beta1" – financial_physician Feb 26 '20 at 06:32
  • @financial_physician Yeah it's a long time :). $x_i$ is an observation of the independent variable. – TooTone Feb 27 '20 at 01:04
2

I believe the problem in your proof is the step where you take the expected value of the square of $\sum_i (x_i - \bar{x} )\left( u_i -\sum_j \frac{u_j}{n} \right)$. This is of the form $E \left[\left(\sum_i a_i b_i \right)^2 \right]$, where $a_i = x_i -\bar{x}; b_i = u_i -\sum_j \frac{u_j}{n}$. So, upon squaring, we get $E \left[ \sum_{i,j} a_i a_j b_i b_j \right] = \sum_{i,j} a_i a_j E\left[b_i b_j \right]$. Now, from explicit computation, $E\left[b_i b_j \right] = \sigma^2 \left( \delta_{ij} -\frac{1}{n} \right)$, so $E \left[ \sum_{i,j} a_i a_j b_i b_j \right] = \sum_{i,j} a_i a_j \sigma^2 \left( \delta_{ij} -\frac{1}{n} \right) = \sum_i a_i^2 \sigma^2$ as $\sum_i a_i = 0$.

Stefano
  • 21
  • 1
2

Begin from "The derivation is as follow:" The 7th "=" is wrong.

Because

$\sum_i (x_i - \bar{x})(u_i - \bar{u})$

$ = \sum_i (x_i - \bar{x})u_i - \sum_i (x_i - \bar{x}) \bar{u}$

$ = \sum_i (x_i - \bar{x})u_i - \bar{u} \sum_i (x_i - \bar{x})$

$ = \sum_i (x_i - \bar{x})u_i - \bar{u} (\sum_i{x_i} -n \bar{x})$

$ = \sum_i (x_i - \bar{x})u_i - \bar{u} (\sum_i{x_i} -\sum_i{x_i})$

$ = \sum_i (x_i - \bar{x})u_i - \bar{u} 0$

$ = \sum_i (x_i - \bar{x})u_i$

So after 7th "=" it should be:

$\frac {1} {(\sum_i(x_i-\bar{x})^2)^2}E\left[\left(\sum_i(x_i-\bar{x})u_i\right)^2\right]$

$=\frac {1} {(\sum_i(x_i-\bar{x})^2)^2}E\left(\sum_i(x_i-\bar{x})^2u_i^2 + 2\sum_{i\ne j}(x_i-\bar{x})(x_j-\bar{x})u_iu_j\right)$

=$\frac {1} {(\sum_i(x_i-\bar{x})^2)^2}E\left(\sum_i(x_i-\bar{x})^2u_i^2\right) + 2E\left(\sum_{i\ne j}(x_i-\bar{x})(x_j-\bar{x})u_iu_j\right)$

=$\frac {1} {(\sum_i(x_i-\bar{x})^2)^2}E\left(\sum_i(x_i-\bar{x})^2u_i^2\right) $, because $u_i$ and $u_j$ are independent and mean 0, so $E(u_iu_j) =0$

=$\frac {1} {(\sum_i(x_i-\bar{x})^2)^2}\left(\sum_i(x_i-\bar{x})^2E(u_i^2)\right) $

$\frac {\sigma^2} {(\sum_i(x_i-\bar{x})^2)^2}$

user158565
  • 7,032
  • 2
  • 9
  • 19
  • 1
    It might be helpful if you edited your answer to include the correct line. – mdewey Apr 24 '17 at 07:48
  • Your answer is being automatically flagged as low quality because it's very short. Please consider expanding on your answer – Glen_b Apr 24 '17 at 12:04