1

In case of perfect multicollinearity the predictor matrix is singular and therefore cannot be inverted . Under these circumstances, the ordinary least-squares estimator $\hat\beta=(\Bbb X'\Bbb X)^{-1}\Bbb X'\Bbb y$ does not exist (Wikipedia) .

I can't visualize the situation.

  • When does the situation of perfect multicollinearity occur ?

  • In case of perfect multicollinearity, why is the predictor matrix singular ?

  • Under these circumstances, why does the ordinary least-squares estimator $\hat\beta=(\Bbb X'\Bbb X)^{-1}\Bbb X'\Bbb y$ not exist ?

ABC
  • 1,367
  • 3
  • 13
  • 31
  • You may find my answer [here](http://stats.stackexchange.com/questions/139593/why-multiple-linear-regression-cannot-be-built-when-pn/139599#139599) helpful for your third bullet point. – TrynnaDoStat Mar 29 '15 at 15:07

2 Answers2

9
  • The situation of perfect multicollinearity occurs when one (or more) column(s) of the design matrix are linear combinations of the others, i.e. the set of column vectors of the matrix $\mathbf{X}$ is linearly dependent.
  • As you might recall from your linear algebra course, if a matrix has size $n \times p$, with $p < n$, the maximum rank of $\mathbf{X}$ will be $p$. As the columns are not independent the rank will be lower than $p$. However, to be formal, the terms singular / non singular refer only to square matrices. In this case you should say that $\mathbf{X}$ is not full column rank.
  • You can see that you need to invert the matrix $\mathbf{X}'\mathbf{X} \in \mathbb{R}^{p\times p}$, for which the property $\mathrm{rk}(\mathbf{X}) = \mathrm{rk}(\mathbf{X}'\mathbf{X})$ holds. It is clear that it can't be inverted. In fact $\mathbf{X}'\mathbf{X}$ will be nonsingular only if its rank equals $p$, but as $\mathrm{rk}(\mathbf{X}) < p$ it will be singular. In the case of the matrix $\mathbf{X}$ having rank $p < n$, i.e. having dependent columns, the system of normal equations $\mathbf{X}'\mathbf{X} \boldsymbol{\beta} = \mathbf{X}'\mathbf{y}$ will have infinite solutions depending on $n-p$ variables. There are infinitely many vectors that solve our initial minimisation problem: $$ \min_{\boldsymbol{\beta}} \| \mathbf{y} - \mathbf{X}\boldsymbol{\beta} \|.$$ You can choose, for example, the one having minimum norm.
ancillary
  • 512
  • 3
  • 8
9

It might be helpful to add that in this situation there will be infinitely many least squares solutions. The problem is not that a least squares solution does not exist, but rather that the least squares solution is not unique.

The formula using $(X'X)^{-1}$ isn't applicable because the inverse matrix doesn't exist. However, any $\beta$ that satisfies the normal equations $X'X\beta=X'y$ (there will be infinitely many of these solutions) is a least squares solution.

Brian Borchers
  • 5,015
  • 1
  • 18
  • 27
  • If i get many $\hat\beta$ in such circumstance, do all of them unbiased estimate of $\beta$ ? – ABC Mar 29 '15 at 17:14
  • 2
    You have to start by specifying which of the least squares solutions you want to consider. For example, you might pick the solution that minimizes $\| \hat{\beta} \|$ among all of the least squares solutions. Unfortunately, this isn't an unbiased estimator. – Brian Borchers Mar 29 '15 at 17:21