3

I have been looking into ridge regression as a method to address multicollinearity in data.

I am aware that multicollinearity can cause high variance in coefficient estimates. I have seen equations such as this:

$var(\hat{\beta}) = \sigma^2(X'X)^{-1}$

I have read that when perfect multicollinearity is present, the matrix is singular and hence no inverse exists. When multicollinearity is present (but not perfect multicollinearity) than the matrix becomes ill-conditioned. This apparently causes the $(X'X)^{-1}$ term to become very large, inflating the variance of $\beta$.

Seeing as the condition score of a matrix is the ratio is $ \sqrt{\frac{\lambda_{max}}{\lambda_{min}}}$ this suggests that multicollinearity causes a larger difference between the eigenvalues of $X'X$.

Based on the above i have 2 questions:

1) Why, when $X'X$ is ill-conditioned, does $(X'X)^{-1}$ become very large?

2) Please can you explain how multicollinearity cause the eigenvalues of X'X to change, as well as why there is a greater difference in their magnitudes between eachother?

Sean
  • 614
  • 2
  • 12

1 Answers1

3
  1. Because the inverse of a small number is large. The inverse of a Grammian matrix $K = Q\Lambda Q^T$ where $Q$ is the eigenvectors matrix and $\Lambda$ the eigenvalue matrix, is effectively the $K^{-1} = Q\Lambda^{-1} Q^T$. As such when we inverse a very small eigenvalue from the diagonal matrix $\Lambda$, we get a very large number in the inverse of it as well as subsequently on the $K^{-1}$. Wikipedia commonly is great for such topics so checking the section: Matrix inverse via eigendecomposition is a good first step to get some further background.
  2. The multicollinearity is caused by a linear dependence between between the columns of $X$. In that sense we already had a problem with $X$ just this was highlighted in $X^TX$. Note that by taking $X^TX$ we are squaring its respective eigenvalues (if $X$ was square) or its respective singular values (in a more general case); the square of 0 is still 0 and the square of a number less than 1 is something even smaller.
  3. For the multicollinearity itself: it means that despite having a $p$ dimensional data ($p$ being our number of features), the data in our design matrix contains enough information for $q < p$ dimensions. For example, think of us having both imperial (pounds) and metric (kilograms) weight measurements; realistically we have information across a single dimension (weight), not two. Because we only have variance across a single dimension, the variance across the second dimension is zero. As that variance maps directly to the eigenvalues, then we get that zero-th (or very small) eigenvalue. (That is only natural as the eigenvalues of a $X^TX$ matrix are the variances of its of the matrix independent coordinate. Unless you have already read it CV.SE has an epic thread on the matter here: Making sense of principal component analysis, eigenvectors & eigenvalues to assist your understanding of eigenvectors and eigenvalues.)
usεr11852
  • 33,608
  • 2
  • 75
  • 117
  • this makes a lot more sense now, thanks! I've have read that thread previously, but never thought about the link between this topic and principal component analysis. So you're saying the eigenvectors found when performing the eigen decomposition of $Q\Lambda Q' $ are equivalent to the eigenvectors we find when performing PCA? – Sean May 22 '20 at 18:32
  • 1
    Up to a scaling factor, yes. For them to be equivalent we need 1. $X^X$ to be scaled by $\frac{1}{n-1}$ and 2. for the columns of $X$ to be centred to zero. Actually it is quite close, for example the degrees of freedom from ridge regression are $\sum_{i=1}^p \frac{d_i^2}{\lambda + d_i^2}$ where $d_i$ are the diagonals of $D$ from $X= USV^T$ the singular decomposition of $X$. – usεr11852 May 22 '20 at 19:57