4

When dealing with the curse of dimensionality, regularization methods seem to be clear in their intuition. All "regularization" methods can be seen as a "squeezing" of one's variables towards 0, when there exists too many of them.

However, PCA dimensionality reduction is another way to handle the same curse of dimensionality, but it does not seem that PCA possesses any comparable interpretation.

Essentially, does a similar intuitive interpretation exist for PCA as for regularization, which makes the former comparable to the latter?

Any insights placing both frameworks on the same meta-footing would be appreciated.

Coolio2654
  • 642
  • 5
  • 17
  • 1
    PCA regularization is very similar to the ridge regularization. See here: https://stats.stackexchange.com/a/133257/28666 – amoeba Jun 25 '19 at 06:35
  • I've gone through that answer before, and, quite frankly, it is a bit complex to grasp through all the math (which is nice, but not helpful in my context). – Coolio2654 Jun 25 '19 at 06:46

1 Answers1

6

PCA considers only the variance of the features ($X$) but not the relationship between features and labels while doing this compression. Regularization, on the other hand, acts directly on the relationship between features and labels and hence develops models which are better at explaining the labels given the features.

I'm not familiar with other fields but Finance literature, in particular Shrinking the Cross Section, has done this comparison (PCA of features vs regularized model) and found that regularized model does a better job of predicting portfolio returns (better out-of-sample $R^2$)

user2522806
  • 716
  • 4
  • 10