3

I have a matrix of let's say 120 variables and 50 subjects (rows). Before computing correlation between the 120 variables, I want to perform principal copmonent analysis (PCA) on this matrix. I will find 120 eigenvalues. Before sorting them and deciding which ones to keep, I have a problem in interpretation. Do these eigenvalues correspond to the original variables (first eigenvalue -- first variable, etc.)?

So if I reject let's say the 10th eigenvalue, does it mean that the 10th variable (among the 120) must be rejected?

amoeba
  • 93,463
  • 28
  • 275
  • 317
Linda
  • 31
  • 1
  • 2
  • 4
    No, this is not correct. PCA summarizes information across variables, and the way to track the relations between variables and eigenvalues are through eigenvectors / loadings. Also, I don't understand what "rejecting" an eigenvalue means. – StasK Jul 16 '15 at 16:46
  • @Linda, if you have 120 variables and 50 samples, you will only get 49 eigenvalues. PCA does not select variables, it constructs new variables. None of these new 49 variables corresponds to one particular original variable. Kicking out one PCA dimension does not mean kicking out one particular original variable. See also http://stats.stackexchange.com/questions/60888. – amoeba Jul 16 '15 at 22:11
  • On the correspondence between principal components and original variates, there are a number of other relevant posts on site; e.g. see [here](http://stats.stackexchange.com/questions/156116/how-to-apply-regression-on-principal-components-to-predict-an-output-variable) – Glen_b Jul 17 '15 at 00:05
  • Thanks to all of you for the reply. the eigen values don't correspond th the weight of each variable but I am still stuck with my question. If I want to know if A and B (from ) my original data are correlated, how could I answer this question after performing PCA if I loose A and B – Linda Jul 17 '15 at 09:13
  • This sounds like a new question. That's not what you asked. – amoeba Jul 17 '15 at 10:02

2 Answers2

4

Your original data lives in $\cal{R}^{120}$ dimensional space. When you do PCA, you essentially do some change of basis. If you discard nothing, this will be a mere rotation. If you discard one basis, you just eliminate one dimension. The basis vectors corresponding to low eigenvalue are typically the dimensions where is no dynamics (according to variance) and are typically removed. To summarize, PCA does the change of basis (maximum variance directions) and removing low eigenvalue dimensions gets rid of non informative/noisy dimensions in the output. This is effectively dimensionality reduction assuming that the variance captures the dynamics. Jon Shlen's tutorial on PCA gives an excellent example with the spring moving in 3D space [link].

Answering to the original question (thanks, amoeba), the answer is "no". Each eigenvector is a new basis vector. It depends on all dimensions from the original input space. The corresponding eigenvalue just tells how much variance that particular basis explains from the total. The probabilistic PCA formulation can shed more light as the latent variables correspond to the hidden factors which explain the data in variance terms...

Vladislavs Dovgalecs
  • 2,315
  • 15
  • 18
1

The direct answer is a definite no.

PCA created 120 entirely new dimensions. And they do not correspond to the original dimensions individually.

cairo2050
  • 11
  • 3
  • 3
    I don't think its correct to say "created new dimensions". The vector space is the same 120 dimensional one as before the pca, it's only the basis that has changed. – Matthew Drury Jul 16 '15 at 22:36
  • +1, but note that if there is only 50 samples (like in the OP), then only 49 principal components will be found, not 120. – amoeba Jul 16 '15 at 22:39
  • @amoeba Technically all 120 principal components will be found, only 49 of them will have non-zero eigenvalues. The covariance matrix from which we find the eigenvalues/eigenvector pairs is 120x120 and its rank is no more than 49, I guess... – Vladislavs Dovgalecs Jul 16 '15 at 22:43
  • 1
    @xeon That's correct about the size and rank of covariance matrix, so the other 120-49 eigenvectors can be arbitrary orthogonal vectors in the remaining space, they are not constrained by the data at all. I don't think I would call such constant zero projections "principal components". – amoeba Jul 16 '15 at 22:45
  • @amoeba yes, I agree. – Vladislavs Dovgalecs Jul 16 '15 at 22:46