2

A bit of a neophyte question:

I want to conduct data reduction on an NLP dataset 2000+ variables and 100000 plus cases.

I am looking at different data reduction techniques discussed in

"Robust Methods for Data Reduction" by Alessio Farcomeni and Luca Greco

However, I am not quite clear as to what criteria I need to consider when selecting between a "standard" PCA and a sparse PCA methodology.

Any pointers would be appreciated

Jake
  • 129
  • 3

2 Answers2

5

Since there are tractable approximate algorithms for both sparse and dense PCA, you should choose based on which one you think is the best estimator for your situation or which one you think will help you solve your applied problem.

Strengths and weaknesses in application

One important strength of sparse PCA is that it can be easier to interpret. There's more on that topic here:

How exactly is sparse PCA better than PCA?

Strengths and weaknesses as estimators

In a typical PCA, the largest eigenvector of the sample covariance is not a consistent estimator of the largest eigenvector of the true covariance unless $\lim \frac{p}{n}=0.$ If you have too many features and not enough samples, dense PCA is totally misleading. If you have enough samples, this is not a problem. In your situation, it sounds like $\frac{p}{n}$ is small ($\approx 0.02$), so dense PCA seems reasonable. I don't know of any diagnostics to test this, but here's an attempt to make some up.

  • If your features are standardized, you can compute the Marchenko-Pastur upper bound, $(\sqrt{\frac{p}{n}} + 1)^2$. Are your largest eigenvalues bigger than this? If not, they could be pure noise.
  • Compute the PCA twice on multiple disjoint or independently selected subsets of your data. Are the largest components approximately the same on each subset?

Sparsity-based approximations can perform better than dense PCA (they are consistent with fewer samples), if you use a basis in which the true principal components are approximately sparse. You can read more about this here.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2898454/

I don't know of any diagnostics to see whether your data are close enough to meeting the sparsity assumptions. I would fall back on my earlier diagnostic: compute the PCA twice on multiple disjoint or independently selected subsets of your data. Are the largest components approximately the same on each subset?

eric_kernfeld
  • 4,828
  • 1
  • 16
  • 41
2

You probably want a sparse PCA. The SVD algorithm that underlies PCA is $O(N M^2)$, which means that, while the matrix for your problem will just about fit in the memory of a standard PC, performing an SVD on it will take a very long time -- at least hours, more likely days.

Theoretically, you pretty much always want the "real" PCA, but for large problems time and space constraints will make that impractical. When it becomes impractical, you choose one of the sparse, online, or pseudo algorithms meant to approximate it.

David Wright
  • 2,181
  • 12
  • 12
  • Thank you. So the criteria are more memory related rather than model related. That is very helpful. – Jake Sep 29 '17 at 04:39
  • 1
    I disagree with this answer. Sparse PCA is not an approximation to dense PCA meant solely for computational purposes. It's a better estimator for certain situations. It's also a worse estimator for certain situations. The question asks what those situations are and which one applies to them. – eric_kernfeld Jan 29 '19 at 01:20
  • Actually, the question could also be read as asking about interpretability, or about all of these aspects. – eric_kernfeld Jan 29 '19 at 01:42