3

The ridge regression estimate is given by $$\beta^{*}=(X'X+kI)^{-1}X'y, k≥0,$$ where $X$ is the feature matrix. The original paper, Hoerl and Kennard's Ridge Regression: Biased Estimation for Nonorthogonal Problems, states that the eigenvalues, $\lambda_i$, of $X'X$ are related to eigenvalues, $\xi_i$, of $W = (X'X+kI)^{-1}$ as $\xi_i=1/(k+\lambda_i)$. This expression follows from solving the characteristic equation $|W - \xi_iI|=0$. I can only imagine using cofactor representation of the determinant. However, the inverse in $W$ complicates matters.

How exactly does one solve this characteristic equation?

einar
  • 3,258
  • 2
  • 19
  • 31
DharmaBum
  • 33
  • 5

1 Answers1

6

If $\lambda$ is a nonzero eigenvalue of an invertible matrix $A$, then $\lambda^{-1}$ is an eigenvalue of $A^{-1}$. To show this, note that if $Av=\lambda v$ then $v = A^{-1}(\lambda v) = \lambda A^{-1} v$ and thus $\lambda^{-1} v = A^{-1} v$.

angryavian
  • 1,733
  • 12
  • 11
  • Just want to point out that we don't need to assume that $\lambda$ is nonzero if we're already assuming that $A$ is invertible, as all eigenvalues of invertible matrices are nonzero. – Eric Perkerson Oct 04 '20 at 19:53