Finding variance-covariance of $\hat\beta$ from $\hat\beta = (X^TX)^{-1}X^Ty$

Question

I was reading from The Elements of Statistical Learning (Section 3.2) and they were able to find $Var(\hat\beta)$

Given

$$\hat\beta = (X^TX)^{-1}X^Ty \qquad (3.6) $$

The answer they had was $Var(\hat\beta) = (X^TX)^{-1}\sigma^2$

Please how do I find $Var(\hat\beta)$ from $ (3.6)$

gunes · Accepted Answer · 2020-10-23T20:59:41.267

5

Via substitution and some matrix algebra: $$\begin{align}\operatorname{var}(\hat\beta)&=\mathbb E[\hat\beta\hat\beta^T]-\mathbb E[\hat\beta]\mathbb E[\hat \beta^T]\\&=\mathbb E[(X^TX)^{-1}X^Tyy^TX(X^TX)^{-1}]-\mathbb \beta\beta^T\\&=(X^TX)^{-1}X^T\mathbb E[yy^T]X(X^TX)^{-1}-\beta\beta^T\\&=(X^TX)^{-1}X^T(\sigma^2I+X\beta\beta^TX^T)X(X^TX)^{-1}-\beta\beta^T\\&=\sigma^2(X^TX)^{-1}\end{align}$$

Note that since $y=X\beta+\epsilon$, and $\mathbb E[\epsilon\epsilon^T]=\sigma^2I$ and $\mathbb E[\epsilon]=0$: $$\mathbb E[yy^T]=X\beta \beta^TX+\mathbb E[\epsilon\epsilon^T]=X\beta \beta^TX+\sigma^2I$$

edited Oct 23 '20 at 20:59

answered Oct 23 '20 at 18:42

gunes

49,700
3
39
75

4

The algebra would become less tedious if using the formula for the variance matrix of linear transformations of random vectors $\operatorname{Var}(Ay)=A\operatorname{Var}(y)A^T$ – Jarle Tufto Oct 23 '20 at 18:50
@gunnes does it mean $var(\hat\beta) = E[\hat\beta\hat\beta^T] - E[\hat\beta]E[\hat\beta^T]$ – EA Lehn Oct 23 '20 at 19:09
@EALehn yes, that is the definition of covariance for random vectors – gunes Oct 23 '20 at 19:11
@gunnes why is the $E[(X^TX)^{-1}X^Ty] = (X^TX)^{-1}X^T E[y] $ – EA Lehn Oct 23 '20 at 19:29
1

Because only randomness is inside $y$ – gunes Oct 23 '20 at 19:40
@gunnes I don't understand the last but one step. Also what is $I$ in $\sigma^2I$ – EA Lehn Oct 23 '20 at 19:55
If you distribute every term, only the last line stays. I means identity – gunes Oct 23 '20 at 20:06
@JarleTufto agreed, that'd look nicer. – gunes Oct 23 '20 at 22:14

Finding variance-covariance of $\hat\beta$ from $\hat\beta = (X^TX)^{-1}X^Ty$

1 Answers1