16

I am trying to understand Principal Component Analysis (PCA). I found a webpage on PCA that introduces it and the concept of the percentage of variance. However, I am very confused about what "Percentage of Variance" (POV) means. Here are the questions I have:

  1. Is there a formal definition or mathematics formula to define "percentage of variance"?
  2. Do we only use POV for calculating PCA? Is POV used somewhere else?
  3. Is POV the same as "Proportion of Variance Explained"? (There are many similar terms online which really confuse me.)
Nick Cox
  • 48,377
  • 8
  • 110
  • 156
Cassie
  • 265
  • 1
  • 2
  • 8
  • Cassie, PCA works with variance-covariance matrix, where variances are its diagonal. Sum of variances is overall multivariate variability. It is this POV is computed of. For more, see [here](http://stats.stackexchange.com/questions/22569/pca-and-proportion-of-variance-explained/22571#22571) – ttnphns Jul 09 '12 at 06:52
  • The link you provided is great. Thus, intuitively, when a component has accounted for larger percentage of variance, can I say the component is more important then others ? Moreover, assume I develop a new variable selection method to select only 5 out of 10 original variables. Without using PCA to find eigenvectors, assume I just calculate cov matrix for these 5 variables. Then, can I calculate POV of these 5 variables in the same way ? Or POV can only be applied on eigenvalues ? Thanks, – Cassie Jul 10 '12 at 01:40
  • Using PCA for feature selection is a little bit different, because the eigenvectors usually do not correspond to input features but rather to a linear combination of these features. Therefore, you can use PCA to reduce the dimensionality of your dataset while still preserving most of the variance by creating a new feature space based on the top $k$ eigenvectors (according to their eigenvalues). However, the "new" features will be different from the original ones. – sebp Jul 10 '12 at 08:17

1 Answers1

16

I assume they are referring to the eigenvalues of the corresponding eigenvectors. The eigenvalues in PCA tell you how much variance can be explained by its associated eigenvector. Therefore, the highest eigenvalue indicates the highest variance in the data was observed in the direction of its eigenvector. Accordingly, if you take all eigenvectors together, you can explain all the variance in the data sample. Instead of using the absolute value of variance explained, as indicated by the eigenvalue, you can also get relative numbers by first summing up all eigenvalues and then divide an eigenvalue $\lambda_i$ by this sum $$ \frac{\lambda_i}{\sum_{i=1}^n \lambda_i} . $$ This way you end up with a "percentage of variance" for each eigenvector.

sebp
  • 1,787
  • 13
  • 24
  • 4
    Just a clarification: if $X_1,\ldots,X_p$ are the original random variables, then $\sum_{i=1}^p\lambda_i=\sum_{i=1}^p{\rm Var}(X_i)=tr(\Sigma)$ where $\Sigma$ is the covariance matrix. – MånsT Jul 09 '12 at 08:51
  • Thank you all, these replies explained well. I have added just a small question to clarify further and commented above. Basically, I would like to know if the calculation of POV can be applied on original variables without finding the eigenvalues first. Any idea is welcome. – Cassie Jul 10 '12 at 01:43