7

Suppose that I have a data set of three variables, Calcium, Iron, and Uranium.

Suppose also that I run PCA and obtain the following principal components:

$$\begin{array}{cccc}&PC_1&PC_2&PC_3\\Calcium&0.6729&0.1021&-0.6771\\Iron&0.5331&0.2554&0.5402\\Uranium&0.1123&-0.8007&-0.0432\end{array}$$

The first PC shows Calcium as having the largest importance and Iron as being the second highest correlation. The second PC shows Uranium as having the largest correlation. But, the third PC then again denotes Calcium as having the largest correlation with the response, then Iron second.

My main question is how such a PCA outcome can be interpreted. It makes no sense to say that Calcium is the most explanatory of the variance, as well as also being the third most explanatory variable for the variance.

whuber
  • 281,159
  • 54
  • 637
  • 1,101
Vladhagen
  • 207
  • 3
  • 15
  • 6
    "Explaining" variance may be a poor metaphor. You might interpret PC1 as being the *total* Ca and Fe values, while PC3 is the *difference* between Fe and Ca. From that point of view it should be no surprise that one or more of the original variables can appear with large coefficients in more than one principal component. Some people "rotate" their principal components--that is, form combinations of them in limited ways--in order to isolate the original variables, hopefully improving interpretability. For instance, here PC1+PC3 is roughly $(0,1,0)$ while PC1-PC3 is roughly $(1,0,0)$. – whuber May 16 '17 at 17:56

3 Answers3

18

Your interpretation of PCA components is not correct.

PCA does not tell you which variables account for the most variation in the data, so a statement like

Calcium is the most explanatory of the variance, as well as also being the third most explanatory variable for the variance.

cannot be drawn from a PC analysis.

What it does say is that the direction determined by the vector

$$\begin{array}{cccc}&PC_1\\Calcium&0.6729\\Iron&0.5331\\Uranium&0.1123\end{array}$$

accounts for the most variation in the data. This direction is a combination of the directions determined by the individual variables. This mixing of directions is fundamental to PCA, and it cannot be undone or ignored.

The further principal components are interpreted iteravely, they account for the most variation in the data in directions that are orthogonal to the previous PC directions.

Matthew Drury
  • 33,314
  • 2
  • 101
  • 132
  • I would avoid using the word recursively. Eigenvalue calculation is solving `A-lambda*I =0` and the vectors are determined by solving `A*v = lambda*v`. The greatest variation is determined by the largest absolute eigenvalue – Mohammad Athar May 16 '17 at 16:57
  • I agree that you don't *solve* for the principal components recursively (I think we probably use the QR algorithm), but I do think it is a productive way to understand them conceptually. – Matthew Drury May 16 '17 at 18:05
  • You might simply replace "recursively" by "iteratively," which is a little more correct and seems to eliminate @Mohammad's objection. BTW, did you notice that the vectors given in the question are not unit vectors? – whuber May 16 '17 at 18:14
  • @whuber Good point. And, no, I didn't... I didn't check, which is embarrassing. – Matthew Drury May 16 '17 at 18:33
  • 4
    No need to be embarrassed--I hadn't noticed it either until I read your post and realized the coefficients had to be too small. I am guessing we have been shown only the first three components of higher-dimensional unit vectors. – whuber May 16 '17 at 18:34
  • 3
    @whuber As you noted, I have only included a portion of the vectors so that the problem at hand can be examined without unnecessary tangent details. – Vladhagen May 16 '17 at 19:47
6

You aren't interpreting PCA correctly. PCA finds a whole new basis for your data. It's analogous to a change of basis: https://www.math.hmc.edu/calculus/tutorials/changebasis/ but we choose a particular basis

The new basis is not arbitrary: the vectors are selected based on how much variation they account for. That is to say, PC1 "points in the direction of greatest variability"

Just because the primary component (vector projection) of PC1 and PC3 are in the direction of calcium, we can not say that calcium is the most "important" (whatever that may mean!).

Geeking out about linear algebra:

By the laws of linear algebra, all principal components are orthogonal to each other, and the the amount of explained variance for any given eigvenvalue, E_p is E_p/(sum(E_i) where sum(E_i) is the sum of all eigenvalues

lastly, here's a good discussion on PCA: Making sense of principal component analysis, eigenvectors & eigenvalues

Mohammad Athar
  • 281
  • 1
  • 11
  • 1
    A little quibble: it's potentially confusing to refer first to the "absolute value of the eigenvalues," strongly suggesting they could be negative (thereby indicating you're working with some kind of square root of a matrix) and then claiming the "amount of explained variance" is proportional to each eigenvalue, which implies they *cannot* be negative. You might try to be clearer about what form of PCA you're talking about and what those eigenvalues actually represent. – whuber May 16 '17 at 18:11
0

Correlation is not the same as linear combination with largest variance, which is what PCA finds.

Also the eigenvectors have no particular direction. You can multiply them with $-1$ and those vectors will also be eigenvectors with the same eigenvalue (variance) and then you would get positive $+0.677\cdots$ for third component.

If you want correlation maybe you could check out Canonical Correlation Analysis (CCA) instead.

Nick Cox
  • 48,377
  • 8
  • 110
  • 156
mathreadler
  • 158
  • 7