Automatically fixing ill-conditioning or collinearity

Question

I'm backtesting a regression model, which entails running it on a bunch of bootstrap samples of a "rewound" version of our data set. Unfortunately, in some of these resamplings, I end up getting some "coincidental" dependencies between covariates that result in a covariate matrix that's either rank-deficient or ill-conditioned (on that specific bootstrap sample; the full covariate matrix is not ill-conditioned). Obviously this causes difficulties for inference.

Is there a standard way of solving this automatically/algorithmically? If I were only fitting on one dataset, I'd simply drop one of the features causing the rank deficiency/ill-conditioning, but that's impractical to do for 20 bootstrap samples of 12 different time snapshots.

My default plan is just to switch to penalized regression, but I'm wondering if there's a more principled approach, since I have enough data that I definitely don't need the penalty otherwise.

Answer 1

If you cannot use your original matrix $K$, you might want to use a $K^{\prime}$ that is the closest you can get in terms of Frobenius norm. To do that find the smallest eigenvalues and set them to a higher number positive number. This will give you a new P(S)D matrix that is closer than any other matrix in Frobenius norm, see N.J. Higham 1988 for details.

You can also consider using Tikhonov regularization. In that case you essentially do a ridge regression-like correction. As you original matrix $K$ will become $K^{\prime} = K + \lambda I$, you will add $\lambda$ to all your eigenvalues. While a bit simplistic this choice has a very nice statistical interpretation: you practically say that all your samples are bit ($\lambda$) more noisy than before (and because as you add things along the diagonal no covariances are affected).

I do not touch at all at alternating projection methods for this matter. You can read more about it on this blog-post by Higham; it offers a nice discussion, as well as links to software implement these procedures in R, MATLAB, Python.

It goes without saying that these methods change the condition number of the matrix you are examining. In most cases the new matrix will have a lower condition number as the ratio of the largest singular value of that matrix to the smallest singular value will get smaller, this is not guaranteed to happen though.