4

In a standard regression problem

\begin{equation} \mathbf{y} = \mathbf{X} \beta + \mathbf{e} \ , \end{equation}

the solution to $\beta$ when the system is overdetermined is $\hat{\beta}= \left(\mathbf{X}\Sigma^{-1}\mathbf{X}\right)^{-1}\mathbf{X}^T\Sigma^{-1}\mathbf{y}$, where $\Sigma = \sigma^{-2} \operatorname{Var}(\mathbf{e})$.

The solution to an underdetermined system when $\Sigma\equiv \mathbf{I}$ is $\hat{\beta} = \mathbf{X}^T\left(\mathbf{X}\mathbf{X}^T\right)^{-1}\mathbf{y}$. Can a similar formula to the least squares case be written when $\Sigma \ne \mathbf{I}$ such that $\hat{\beta} = \mathbf{X}^T\Sigma^{-1}\left(\mathbf{X}\mathbf{X}^T\Sigma^{-1}\right)^{-1} \mathbf{y}$?

I have not been able to find any good references on this system and would appreciate if you could suggest any also.

hatmatrix
  • 739
  • 5
  • 15
  • 1
    Could you please clarify what you are asking? I do not see any "formulations" in your question--only *formulas*--so the meaning of "similar formulation to the least squares case" is not apparent. – whuber Jan 20 '14 at 16:30
  • My apologies. I meant formulation in the construction of a formula but did not mean anything more abstract. I have changed the the word to say "formula" rather than "formulation." – hatmatrix Jan 20 '14 at 16:51
  • 2
    Note that your expression for $\hat{\beta}$ in the line starting with "The solution to..." is incorrect; you need to post-multiply by $y$. – jbowman Jan 20 '14 at 21:47

1 Answers1

4

Yes, it can be derived in the same way as the Generalized Least Squares estimator.

To customize the result to an underdetermined system, note the derivation of GLS as the best linear unbiased estimator uses the factorization $\Sigma = AA^T$ as follows. Define $Y^* = A^{-1}Y$, $X^*=A^{-1}X$, and $e^*=A^{-1}e$. Then the OLS solution to $Y^* = X^*\beta + e^*$ satisfies the Gauss-Markov requirements, and is consequently the best linear unbiased estimator.

When using the right generalized inverse $C^+ = C^T(CC^T)^{-1}$ to form the estimate $\hat{\beta} = X^+Y$ in the underdetermined case, you can just use $X^*$ and $Y^*$ instead of $X$ and $Y$. Writing it out in terms of $A$ and $X$ gives:

$\hat{\beta} = (A^{-1}X)^T(A^{-1}XX^T(A^{-1})^T)^{-1}A^{-1}Y$

jbowman
  • 31,550
  • 8
  • 54
  • 107