4

The multiple linear regression model is given by $$ \mathbf{y} = \mathbf{X} \mathbf{\beta} + \mathbf{\epsilon} \\ \mathbf{\epsilon} \sim N(0, \sigma^2 \mathbf{I}) $$

It is known that an estimate of $\mathbf{\beta}$ can be written as $$ \hat{\mathbf{\beta}} = (\mathbf{X}^{\prime} \mathbf{X})^{-1}\mathbf{X}^{\prime} \mathbf{y} $$

Hence $$ \textrm{Var}(\hat{\mathbf{\beta}}) = (\mathbf{X}^{\prime} \mathbf{X})^{-1} \mathbf{X}^{\prime} \; \sigma^2 \mathbf{I} \; \mathbf{X} (\mathbf{X}^{\prime} \mathbf{X})^{-1} = \sigma^2 (\mathbf{X}^{\prime} \mathbf{X})^{-1} $$

Let $\mathbf{x}_j$ be the $j^{th}$ column of $\mathbf{X}$, and $\mathbf{X}_{-j}$ be the $\mathbf{X}$ matrix with the $j^{th}$ column removed.

I am told that the variance of $\hat{\mathbf{\beta}}_j$ can therefore be expressed as $$ \textrm{Var}(\hat{\mathbf{\beta}}_j) = \sigma^2 [\mathbf{x}_j^{\prime} \mathbf{x}_j - \mathbf{x}_j^{\prime} \mathbf{X}_{-j} (\mathbf{X}_{-j}^{\prime} \mathbf{X}_{-j})^{-1} \mathbf{X}_{-j}^{\prime} \mathbf{x}_j]^{-1} $$

Can anyone shed some light on how to prove it?

Hints would suffice.

  • Are you sure $X_{-j}$ should not be $X$ with the $j$th *column* removed? – Christoph Hanck Oct 31 '16 at 10:21
  • *Multilinear*? I think the term *multiple* means the same and is much more common (20 times as popular according to Google search). – Richard Hardy Oct 31 '16 at 10:34
  • In the model as written above, the rows of the matrix $X$ represent observations, not regressors. So it should be about removing a _column_ of $X$, not a row (but maybe it was meant a row of $X'$?). . The expression with the $j$-th _observation_ removed is certainly not valid. – Alecos Papadopoulos Oct 31 '16 at 17:21

1 Answers1

2

Let $\mathbf{x_1}$ be the $1$st column of $X$. Let $X_{-1}$ be the matrix $X$ with the $1$st column removed.

Consider the matrices: \begin{align*} A &= \mathbf{x_1}'\mathbf{x_1}\quad \quad &\text{1 by 1 matrix}\\ B &= \mathbf{x_1}'X_{-1} \quad &\text{1 by n-1 matrix}\\ C &= X_{-1}\mathbf{x_1} & \text{n-1 by 1 matrix} \\ D &= X_{-1}'X_{-1} & \text{n-1 by n-1 matrix} \end{align*}

Observe that:

$$X'X = \begin{bmatrix}A & B \\C & D \end{bmatrix}$$

By the matrix inversion lemma (and under some existence conditions):

$$\left(X'X \right)^{-1} = \begin{bmatrix}\left(A - BD^{-1}C \right)^{-1} & \ldots \\ \ldots & \ldots \end{bmatrix}$$

Notice the 1st row, 1st column of $(X'X)^{-1}$ is given by the Schur complement of block $D$ of the matrix $X'X$

$$\left(A - BD^{-1}C \right)^{-1}$$

Matthew Gunn
  • 20,541
  • 1
  • 47
  • 85