2

In looking over TooTone's answer in Derive Variance of regression coefficient in simple linear regression, there's a step in line 3 below where $(\beta_0 + \beta_1x_i + u_i )$ is simplified to $u_i$ on the basis that only $u_i$ is a random variable. I'm having trouble understanding why the $\beta_0 + \beta_1x_i$ terms can be ignored as a result of this? Is anyone able to offer more detail on either a reason or an intermediate step that I'm not following?

\begin{align} \text{Var}(\hat{\beta_1}) & = \text{Var} \left(\frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} \right) \\ &= \text{Var} \left(\frac{\sum_i (x_i - \bar{x})(\beta_0 + \beta_1x_i + u_i )}{\sum_i (x_i - \bar{x})^2} \right), \;\;\;\text{substituting in the above} \\ &= \text{Var} \left(\frac{\sum_i (x_i - \bar{x})u_i}{\sum_i (x_i - \bar{x})^2} \right), \;\;\;\text{noting only $u_i$ is a random variable} \\ &= \frac{\sum_i (x_i - \bar{x})^2\text{Var}(u_i)}{\left(\sum_i (x_i - \bar{x})^2\right)^2} , \;\;\;\text{independence of } u_i \text{ and, Var}(kX)=k^2\text{Var}(X) \\ &= \frac{\sigma^2}{\sum_i (x_i - \bar{x})^2} \\ \end{align}

Julian Drago
  • 271
  • 2
  • 8
  • When $U$ is random and $x$ is a constant, $\operatorname{Var}(xU)=x^2\operatorname{Var}(U).$ See https://stats.stackexchange.com/search?q=variance+linear+combination+score%3A1+-PCA+-Principal+-stochastic+-process+-regression+self-study. – whuber Jun 02 '19 at 19:06
  • I guess I'm having trouble seeing how $\sum_i (x_i - \bar{x})(\beta_0 + \beta_1x_i + u_i )$ meets $Var(xU)$? – Julian Drago Jun 02 '19 at 19:35
  • Check out the search I gave, then, which links to posts that express the basic properties of variance. – whuber Jun 02 '19 at 19:49
  • I ended up turning to wikipedia, but I guess the piece I was missing was that $Var(X + a) = Var(X)$. – Julian Drago Jun 02 '19 at 20:13

2 Answers2

2

If $X$ and $Y$ are independent random variables, then $$ \textrm{Var}(X+Y)=\textrm{Var} X+\textrm{Var} Y.\qquad (1) $$ This is because $$ \mathbb E\left[(X+Y)^2\right]=\mathbb E[X^2]+2\mathbb E[XY]+\mathbb E[Y^2]\overset{indep!}=\mathbb E[X^2]+2\mathbb EX\mathbb EY+\mathbb E[Y^2]=\left(\mathbb EX+\mathbb EY\right)^2. $$ Similarly if $a$ is deterministic and $X$ is random, then $$\textrm{Var}(aX)=a^2\textrm{Var}X,\qquad (2)$$ since $$ \mathbb E[(aX)^2]=\mathbb E[a^2X^2]=a^2\mathbb E[X^2]. $$ Finally, if $a$ is deterministic and $X$ is random, then $$\textrm{Var}(a+X)=\textrm{Var}X,\qquad (3)$$ since $$ \mathbb E\left[a+X-\mathbb E(a+X)\right]^2=\mathbb E\left[X-\mathbb EX\right]^2. $$

We can use these three properties to answer your question. You have a sequence of independent random variables $(u_i)$ and sequences of constants $(a_i),(b_i)$ where $$a_i=\frac{x_i-\overline{x_i}}{\sum_i(x_i-\overline{x_i})^2},\qquad b_i=\beta_0+\beta_1 x_i$$ and you wish to simplify the expression $$ \textrm{Var}\left(\sum_i a_i(b_i+u_i)\right). $$ Using the third property, $$ \textrm{Var}\left(\sum_i a_i(b_i+u_i)\right)=\textrm{Var}\left(\sum_i a_iu_i\right). $$ Using the first property and independence, $$ \textrm{Var}\left(\sum_i a_iu_i\right)=\sum_i \textrm{Var}(a_iu_i). $$ Using the second property, $$ \sum_i \textrm{Var}(a_iu_i)=\sum_i a_i^2\textrm{Var}u_i. $$ Altogether, this fills in the missing reasoning.

pre-kidney
  • 316
  • 1
  • 4
1

The property used here is that $\mathbb{Var}(\text{const}+X) = \mathbb{Var}(X)$. If you expand the terms out formally, from the second step to the third, you get:

$$\begin{equation} \begin{aligned} \mathbb{Var} (\hat{\beta}_1) &= \mathbb{Var} \left(\frac{\sum_i (x_i - \bar{x})(\beta_0 + \beta_1 x_i + u_i )}{\sum_i (x_i - \bar{x})^2} \right) \\[6pt] &= \mathbb{Var} \left(\frac{\sum_i (x_i - \bar{x})(\beta_0 + \beta_1 x_i) }{\sum_i (x_i - \bar{x})^2} + \frac{\sum_i (x_i - \bar{x}) u_i}{\sum_i (x_i - \bar{x})^2} \right) \\[6pt] &= \mathbb{Var} \left( \text{const} + \frac{\sum_i (x_i - \bar{x}) u_i}{\sum_i (x_i - \bar{x})^2} \right) \\[6pt] &= \mathbb{Var} \left( \frac{\sum_i (x_i - \bar{x}) u_i}{\sum_i (x_i - \bar{x})^2} \right). \\[6pt] \end{aligned} \end{equation}$$

Ben
  • 91,027
  • 3
  • 150
  • 376