Relationship between ridge regression and PCA regression

Question

I remember having read somewhere on the web a connection between ridge regression (with $\ell_2$ regularization) and PCA regression: while using $\ell_2$-regularized regression with hyperparameter $\lambda$, if $\lambda \to 0$, then the regression is equivalent to removing the PC variable with the smallest eigenvalue.

• Why is this true?
• Does this have anything to do with the optimization procedure? Naively, I would have expected it to be equivalent to OLS.
• Does anybody have a reference for this?

Answer 1

Let $\mathbf X$ be the centered $n \times p$ predictor matrix and consider its singular value decomposition $\mathbf X = \mathbf{USV}^\top$ with $\mathbf S$ being a diagonal matrix with diagonal elements $s_i$.

The fitted values of ordinary least squares (OLS) regression are given by $$\hat {\mathbf y}_\mathrm{OLS} = \mathbf X \beta_\mathrm{OLS} = \mathbf X (\mathbf X^\top \mathbf X)^{-1} \mathbf X^\top \mathbf y = \mathbf U \mathbf U^\top \mathbf y.$$ The fitted values of the ridge regression are given by $$\hat {\mathbf y}_\mathrm{ridge} = \mathbf X \beta_\mathrm{ridge} = \mathbf X (\mathbf X^\top \mathbf X + \lambda \mathbf I)^{-1} \mathbf X^\top \mathbf y = \mathbf U\: \mathrm{diag}\left\{\frac{s_i^2}{s_i^2+\lambda}\right\}\mathbf U^\top \mathbf y.$$ The fitted values of the PCA regression (PCR) with $k$ components are given by $$\hat {\mathbf y}_\mathrm{PCR} = \mathbf X_\mathrm{PCA} \beta_\mathrm{PCR} = \mathbf U\: \mathrm{diag}\left\{1,\ldots, 1, 0, \ldots 0\right\}\mathbf U^\top \mathbf y,$$ where there are $k$ ones followed by zeroes.

From here we can see that:

1. If $\lambda=0$ then $\hat {\mathbf y}_\mathrm{ridge} = \hat {\mathbf y}_\mathrm{OLS}$.

2. If $\lambda>0$ then the larger the singular value $s_i$, the less it will be penalized in ridge regression. Small singular values ($s_i^2 \approx \lambda$ and smaller) are penalized the most.

3. In contrast, in PCA regression, large singular values are kept intact, and the small ones (after certain number $k$) are completely removed. This would correspond to $\lambda=0$ for the first $k$ ones and $\lambda=\infty$ for the rest.

4. This means that ridge regression can be seen as a "smooth version" of PCR.

   (This intuition is useful but does not always hold; e.g. if all $s_i$ are approximately equal, then ridge regression will only be able to penalize all principal components of $\mathbf X$ approximately equally and can strongly differ from PCR).

5. Ridge regression tends to perform better in practice (e.g. to have higher cross-validated performance).

6. Answering now your question specifically: if $\lambda \to 0$, then $\hat {\mathbf y}_\mathrm{ridge} \to \hat {\mathbf y}_\mathrm{OLS}$. I don't see how it can correspond to removing the smallest $s_i$. I think this is wrong.

One good reference is The Elements of Statistical Learning, Section 3.4.1 "Ridge regression".

---

See also this thread: Interpretation of ridge regularization in regression and in particular the answer by @BrianBorchers.

Answer 2

Elements of Statistical Learning has a great discussion on this connection.

The way I interpreted this connection and logic is as follows:

• PCA is a Linear Combination of the Feature Variables, attempting to maximize the variance of the data explained by the new space.
• Data that suffers from multicollinearity (or more predictors than rows of data) leads to a Covariance Matrix that does not have full Rank.
• With this Covariance Matrix, we cannot invert to determine the Least Squares solution; this causes the numerical approximation of the Least Squares Coefficients to blow up to infinity.
• Ridge Regression introduces the penalty Lambda on the Covariance Matrix to allow for matrix inversion and convergence of the LS Coefficients.

The PCA connection is that Ridge Regression is calculating the Linear Combinations of the Features to determine where the multicollinearity is occurring. The Linear Combinations of Features (Principle Component Analysis) with the smallest variance (and hence smaller singular values and smaller eigenvalues in PCA) are the ones penalized the hardest.

Think of it this way; for the Linear Combinations of Features with smallest variance, we have found the Features that are most alike, hence causing the multicollinearity. Since Ridge does not reduce the Feature set, whichever direction this Linear Combination is describing, the original Feature corresponding to that direction is penalized the most.

Answer 3

Consider the linear equation $$ \mathbf X \beta = \mathbf y\,, $$ and the SVD of $\mathbf X$, $$ \mathbf X = \mathbf U \,\mathbf S \,\mathbf V^T, $$ where $\mathbf S = \text{diag}(s_i)$ is the diagonal matrix of singular values.

Ordinary least squares determines the parameter vector $\beta$ as $$ \beta_{OLS} = \mathbf V \,\mathbf S^{-1} \,\mathbf U^T $$ However, this approach fails as soon there is one singular value which is zero (as then the inverse does not exists). Moreover, even if no $s_i$ is excatly zero, numerically small singular values can render the matrix ill-conditioned and lead to a solution which is highly susceptible to errors.

Ridge regression and PCA present two methods to avoid these problems. Ridge regression replaces $\mathbf S^{-1}$ in the above equation for $\beta$ by \begin{align} \mathbf S^{-1}_{\text{ridge}} &= \text{diag}\bigg(\frac{s_i}{s^2_i+\alpha}\bigg),\\ \beta_{\text{ridge}} &= \ \mathbf V \,\mathbf S_{\text{ridge}}^{-1} \,\mathbf U^T \end{align}

PCA replaces $\mathbf S^{-1}$ by \begin{align} \mathbf S^{-1}_{\text{PCA}} &= \text{diag}\bigg(\frac{1}{s_i} \, \theta(s_i-\gamma)\bigg)\,,\\ \beta_{\text{PCA}} &= \ \mathbf V \,\mathbf S_{\text{PCA}}^{-1} \,\mathbf U^T \end{align} wehre $\theta$ is the step function, and $\gamma$ is the threshold parameter.

Both methods thus weaken the impact of subspaces corresponding to small values. PCA does that in a hard way, while the ridge is a smoother approach.

More abstractly, feel free to come up with your own regularization scheme $$ \mathbf S^{-1}_{\text{myReg}} = \text{diag}\big(R(s_i)\big)\,, $$ where $R(x)$ is a function that should approach zero for $x\rightarrow 0$ and $R(x)\rightarrow x^{-1}$ for $x$ large. But remember, there's no free lunch.