Making the mindset transition - projection in OLS regression

Question

For example, the space spanned by the columns in

$$\mathbf{X} = \begin{bmatrix} 0 & 0 \\ 1 & 0 \\ 0 & 1 \end{bmatrix}$$

is the y-z plane. Further,

$$\mathbf{X'X} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ \end{bmatrix}$$

$$\mathbf{P = X(X'X)^{-1}X'} = \begin{bmatrix} 0 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$$

$$\mathbf{y} = \begin{bmatrix} 1 \\ 2 \\ 1 \end{bmatrix}$$

Then to project y onto the column space of X,

$$\mathbf{Py} = \begin{bmatrix} 0 \\ 2 \\ 1 \end{bmatrix}$$

I understand why this works. What I don't understand is how this works for linear regression - in the above example, each row represents a dimension in space I think (row 1 - x, row 2 - y, etc). In linear regression, the columns act as the dimensions. How does this transition work?

Thank you for any help!

Answer 1

Given OLS model: $y = X \beta + \epsilon$,

We wish to find the $\beta$ that minimizes the sum of squares error,

$L(\beta) = (y-X\beta)^T(y-X\beta)$

Differentiating $L$ with respect to $\beta$, setting the gradient to zero and solving for $\beta$:

$\nabla_{\beta} L(\beta) = 2X^T(y - X\beta) = 0$

We get that the OLS estimate of $\beta$ is: $\hat{\beta} = (X^TX)^{-1}X^T{y}$

The predicted response, $\hat{y} = X\hat{\beta} = X(X^TX)^{-1}X^T{y} = Py$ where $P = X(X^TX)^{-1}X^T$ is the projection matrix.