4

In Elements of Statistical Learning, pg 47, at the very bottom, it states that $\hat{\beta}$ and $\hat{\sigma}^2$ are statistically independent.

Is this saying that they are independent when conditioned on $X$, or is it saying they're marginally independent? If it is saying they're marginally independent, which seems to be the case in the book, I'm confused because it would seem they're not marginally independent.

We know that $$ \hat{\beta} = (X^TX)^{-1}X^Ty$$

$$\hat{\sigma}^2 = \frac{1}{N - p - 1} ||y - \hat{y} ||_2^2 = \frac{1}{N - p - 1} ||y - X\hat{\beta} ||_2^2 $$

By inspection, $\hat{\sigma}^2$ is a function of $\hat{\beta}$, so it's not obvious how the two are marginally independent.

student010101
  • 334
  • 2
  • 10
  • 1
    Hi: Here's a proof but it's not trivial. https://stats.stackexchange.com/questions/286179/why-is-a-t-distribution-used-for-hypothesis-testing-a-linear-regression-coeffici – mlofton May 10 '21 at 14:16
  • 1
    See https://stats.stackexchange.com/q/173396/119261 or https://stats.stackexchange.com/q/362590/119261. – StubbornAtom May 10 '21 at 14:58
  • 1
    For additional demonstrations, see posts about [Cochran's Theorem](https://stats.stackexchange.com/search?q=cochran%27s+theorem). – whuber May 10 '21 at 14:59
  • @StubbornAtom I looked at the first link, and I am confused how the accepted answer shows this. It shows that the residuals and $\hat{\beta}$ are uncorrelated, but how do we know that $\hat{\beta}$ and the residuals are jointly normal? I think that step is missing. I know that once we show that two variables are jointly normal, then uncorrelated implies independence. – student010101 May 10 '21 at 15:39
  • @StubbornAtom I think I understand now. We need to show that any linear combination of the residuals and $\hat{\beta}$ is normal. The residual and $\hat{\beta}$ are $\epsilon$ scaled plus some constant. So any linear combination of the two is also $\epsilon$ scaled plus some constant. And since we assume $\epsilon$ is normally distributed, any scaling plus shift of it is also normally distributed, so this implies the residuals and $\hat{\beta}$ are jointly normal, hence showing they're uncorrelated means they're independent. – student010101 May 10 '21 at 15:49

0 Answers0