0

I know how to derive β when $X^TX$ has an inverse and on that condition, β is an unbiased estimate of β* with mean 0 and β* and variance $σ^2(X^TX)^{-1}$. But why the variance will become very high when $X^TX$ is singular? Any mathematical proof or explanation is appreciated!

ELI
  • 93
  • 4
  • 4
    When the matrix is singular it doesn't have its inversion. You can't estimate $\beta$ in this case. – Daniel Dostal May 06 '18 at 11:18
  • then how do you know that the variance is very large – ELI May 06 '18 at 11:19
  • 1
    if the design matrix $X$ is close to singular, the estimation of $\beta$ will not be precise, namely it will have high variance. a matrix is close to singular when two columns (for example) are nearly collinear (nearly parallel vectors). this means you have in your model two highly correlated covariates. then your estimation of betas has high variance, "because the model is not able to distinguish well between the two nearly collinear variables". – fabiob May 06 '18 at 11:22
  • Your predictions of new data points are not affected. It is just the ambiguity between the two collinear vectors that generates the uncertainty, but whichever model (with highly varying values for $\beta$) you choose the predictions remain nearly the same. – Sextus Empiricus May 06 '18 at 11:28
  • What do you mean by 'not be able to distinguish' – ELI May 06 '18 at 12:03
  • 1
    In https://stats.stackexchange.com/a/70910/3277 it is gemetrically explained why in almost singular conditions st. errors of b are very high. – ttnphns May 06 '18 at 13:31
  • example: say you wanna estimate the risk for some illness connected with obesity (this will be the dependent variable) and in your design matrix you put BMI, mass of body fat, cholesterol... BMI, cholesterol and fat body mass are correlated (let's assume this at least). so your design matrix X will be close to singular. the model won't be able to distinguish whether an increased risk comes from increased BMI, cholesterol or body fat, as these things are nearly the same thing, and produce the same effect on the illness risk. – fabiob May 06 '18 at 20:23

1 Answers1

2

You can estimate $\beta$ via Psuedoinverse.

Note that you are concerned with variance of $\hat{\beta}$ not $\beta$.

Suppose $X^TX$ is singular. Then There exists one column of x that can be written as a linear combination of the others. Suppose it is the case that column i = column j for some ij. Suppoose the effect of variable i is 10. Then the beta can be estimated by setting i to 0 and giving column j 10, giving column j 0 and i 10, or giving the two any combination between (eg, 5 and 5). Were x singular, column i or j would not be there, say column i is removed, and the remaining j would be 10 always. Hence the variance of column i and j in the singular case is much higher (each varies between 0 and 10).

sjw
  • 5,091
  • 1
  • 21
  • 45