Uniqueness of $x'\beta$ even when $\mathbb{E}(x^Tx)$ is not invertible

Question

As discussed in user25658's answer to this question, when one wants to compute

$$ \beta = \mathbb{E}(x^Tx)^{-1} \mathbb{E}(x^TY) $$

but $\mathbb{E}(x^Tx)$ is not invertible, $\beta$ is not uniquely identified. A characterization of all possible values of $\beta$ is given by

$$\beta= \mathbb{E}(x^Tx)^g\mathbb{E}(x^Ty)$$

where $\mathbb{E}(x^Tx)^g$ is the generalized inverse of $\mathbb{E}(x^Tx)$.

In Hansen, Econometrics, pg 34, section 2.18 (as of today's version on Hansen's webiste) it is written that $x^T\beta = x^T \mathbb{E}(x^Tx)^g\mathbb{E}(x^Ty) $ is however uniquely identified.

This result is not completely counterintuitive to me. I understand that that this is matrix multiplication and that the fact that for every $x$ there may be multiple $\beta(x)$, say $\beta(x)_1 \neq \beta(x)_2$, does not imply $x^T\beta(x)_1 \neq x^T\beta(x)_2$.

Nevertheless, I have been unsuccessful in trying to prove the claim. Could someone help me with that?

Answer 1

Note that when $ E [xx^T] = A$ is not invertible you actually want a solution to the equation

$$ A\beta=E [xx^T]\beta=E [xy]=c$$

Now if a solution exists it must have the form

$$ \hat {\beta}=A^{+}c + (I-A^{+}A)w $$

Where $ w $ is an arbitrary vector with the same dimension as $ c $, and $ A^{+} $ is the moore penrose pseudo inverse - can be calculated by using the spectral decomposition of $ A $. Now what happens to a predicted value? This is given as:

$$ x^T \hat {\beta}=x^TA^{+}c + (x^T-x^TA^{+}A)w $$

Now using the spectral decomposition we have

$$ A=FLF^T\implies A^{+}=FL^{+} F^T $$

This gives us

$$x^T\hat {\beta}=x^TFL^{+} F^Tc + x^TF(I-LL^{+} )F^Tw$$

Now this is basically doing a co-ordinate transform by the matrix $ F $. $ x^{*}=F^Tx $ and $ c^{*}=F^Tc$ and $ w^{*}=F^Tw $. Now we will basically have that

$$x^T\hat {\beta}=\sum_{k=1}^{r}\frac {x^{*}_{k} c^{*}_{k}}{l_k} +\sum_{k=r+1}^{p}x^{*}_{k} w^{*}_{k}$$

Where $ r $ is the rank of $ A=E [xx^T] $ and $ p\geq r $ is the dimension of $ x $. So the arbitrariness of the predictions is "siphoned" off into a $ p-r $ subspace.

To remove it completely we need to plug in the "sample" version of A, namely $$ A_s=\frac {1}{n}\sum_{I=1}^{n} x_ix_i^T $$ and then we note that $ x_i^{*} $ is just the "principal component" score for the ith observation in the sample. This means that $ x_{ik}^{*}=0 $ for $ k> r $. So this means that the arbitrary choice for $ w $ only affects the betas, but not the fitted values.

Note that this is not true for "future" predictions, for $ x $ values that weren't observed in the sample.

Hope this helps.