5

Suppose that you have 1000 features, and a data set made up of say, 50,000 points. Suppose then that we perform PCA, and we extract the top 5 PCs, since they explain 99.99 percent of the variance, and thats all we care about.

From those top 5 PCs, can we 'go backwards', and be able to decipher, what the most 'important' features were from the original 1000? For example, can we answers the question, "What combination of my original 1000 features were responsible for my top PC?"

Thank you.

Creatron
  • 1,407
  • 2
  • 13
  • 23
  • The answer is tied to what exactly you mean by "important". Also, what is "top PC"? PC1 or PC1 through PC5 combined? – ttnphns Sep 14 '13 at 15:16
  • 1
    @ttnphns By top PC I just mean the first one. What I mean by "important": I want to know, what features in my original 1000 feature space, weigh more heavily in giving rise to the first PC. Put another way: I have 1000 features. I found that my first 5 PCs explain 99.99% of the variance. Thus, what features from my original 1000 may I remove? – Creatron Sep 14 '13 at 15:26
  • 1
    The 1st eigenvector values are the regression weights of variables in predicting the 1st component – ttnphns Sep 14 '13 at 16:06
  • @ttnphns Ah yes I think that is what I was looking for thanks. In my googling, I have come across different terminology that may mean the same thing: PCA 'loadings', PCA 'scores', etc. What is the proper terminology for what you just described? Thanks! – Creatron Sep 14 '13 at 16:08
  • That's not different terminology at all, that's different things. Loadings are eigenvector values normalized by the respective eigenvalue. "Scores" are the computed values of a component. – ttnphns Sep 14 '13 at 16:12
  • Maybe the concept you're looking for is "explained variance". See this: http://stats.stackexchange.com/questions/22569/pca-and-proportion-of-variance-explained – Robert Smith Sep 15 '13 at 18:33

3 Answers3

6

Each of the principal components projects the whole original feature space onto several dimensions, which I will call the latent features. The more an original feature contributes to a latent feature, the more important it is for that feature.

Thus, look at the absolute values of the Eigenvectors' components corresponding to the $k$ largest Eigenvalues. The larger they are, the more a specific feature contributes to that principal component.

Mind, however, that these will typically be dense. If you want to find some kind of minimal feature space that explains most of the data, you might be interested in sparse pca.

bayerj
  • 12,735
  • 3
  • 35
  • 56
  • 1
    Thanks that makes a lot of sense. So if I retained only 5 PCs, then I can essentially look at the eigenvectors corresponding to those first 5 eigenvalues, and the absolute values of the eigenvector elements will be the 'importance' so to speak of each original feature. Have I understood you right? Question, must not one normalize the eigenvectors by its corresponding eigenvalue first? – Creatron Sep 14 '13 at 20:15
  • 1
    If you want to compare them over different eigenvalues, that might make sense yes. – bayerj Sep 15 '13 at 13:21
  • @bayerj, must you normalise the vector by its corresponding eigenvalue or normalize to the magnitude of value 1 (unit vector)? – Vass Feb 26 '15 at 14:22
2

You can answer the last question by looking at the loadings of that PC. You could also do this for each of the other 5 PCs. I am not sure what you could do about the overall importance of particular variables.

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
  • You mean the eigen values from the decomposition? Right, of my original 1000 features, I found that my 5 top PCs explain 99.99% of the variance. My question is, which of my original 1000 features can I do away with? Clearly a lot, because I went from 1000 to 5. But which ones? – Creatron Sep 14 '13 at 15:28
  • Not the eigenvalues, the loadings of the variables on the components. Which software are you using (the terminology may differ). – Peter Flom Sep 14 '13 at 15:35
  • Let me ask a different way, I have decomposed the correlation matrix of my data set $C$, into $P \Sigma P^H = C$. How do you ascertain the pc loadings from here? – Creatron Sep 14 '13 at 16:00
  • I don't know.... I know how to find the loadings from `SAS` output or `R` output of PCA. – Peter Flom Sep 14 '13 at 17:07
  • @TheGrapeBeyond: the loadings (aka rotation matrix) are the eigenvectors $P$. The eigenvalues $\Sigma$ give the variance covered by that component. – cbeleites unhappy with SX Oct 07 '13 at 20:24
  • @cbeleites Yes I think that makes sense - thanks! :) – Creatron Oct 08 '13 at 01:02
1

Since eigen vectors are from the linear combination of the original variables, I don't think you can safely determine which of the original variables you can do away with from PCA. I felt this is more like a Factor Analysis question (related to latent variables).

user31180
  • 61
  • 1
  • 1