Derivation of the closed-form solution to minimizing the least-squares cost function

Question

I'm reading the The Elements of Statistical Learning and come across the following:

Can anyone explain how they moved from 3.3 to 3.6 or even to 3.4 ?

Answer 1

Our loss function is $RSS(\beta) = (y - X\beta)^T(y -X\beta)$. Expanding this and using the fact that $(u - v)^T = u^T - v^T$, we have $$ RSS(\beta) = y^Ty - y^TX\beta - \beta^TX^Ty + \beta^T X^T X \beta. $$ Noting that $y^TX\beta$ is a scalar, and for any scalar $r \in \mathbb R$ we have $r = r^T$ we have $y^T X \beta = (y^T X \beta)^T = \beta^T X^T y$ so all together $$ RSS(\beta) = y^T y - 2 \beta^T X^T y + \beta^T X^T X \beta. $$

Now we'll differentiate with respect to $\beta$: $$ \frac{\partial RSS}{\partial \beta} = \frac{\partial}{\partial \beta} y^T y - 2 \frac{\partial}{\partial \beta} \beta^T X^T y + \frac{\partial}{\partial \beta} \beta^T X^T X \beta $$ $$ = 0 - 2 X^T y + 2 X^T X \beta. $$ If you haven't seen derivatives with respect to a vector before, the Matrix Cookbook is a popular reference.

We want to find the minimum of $RSS$ so we'll set the derivative equal to $0$. This leads us to $$ \frac{\partial RSS}{\partial \beta} \stackrel{\text{set}}= 0 \implies -2X^T y + 2X^T X \beta = 0 $$ $$ \implies X^T y - X^T X \beta = 0 \implies X^T(y - X \beta) = 0. $$

Now we do use the assumption that $X$ is full column rank, which means we know $X^T X$ is positive definite and therefore invertible. This means $$ X^Ty = X^T X \beta \implies (X^T X)^{-1}X^T y = \hat \beta $$ where we achieved this by left-multiplying by $(X^T X)^{-1}$.