6

Principal component analysis (PCA) works like this: the first greatest variance on the first principal component, the second greatest variance on the second principal component, and so on.

For me there is a problem with this iterative process.

What if I know I only want two principal components in order to visualize my data in 2 dimensions?

The two first PCs are not the best because the second one is the best 2nd PC but with the 1st PC they don't constitute the best couple of principal components.

Is there a way to find the "best couple" of principal components, meaning two-dimensional subspace with the greatest variance?

amoeba
  • 93,463
  • 28
  • 275
  • 317
Ophelie
  • 383
  • 2
  • 8

3 Answers3

17

While I completely agree with the statistical correctness of Peter Flom's answer (+1), I believe it is worth mentioning that Independent component analysis (ICA) might offer an insightful alternative. ICA provides components that are not constrained to be orthogonal with each other; this means that for some purposes ICs might be more helpful than PCs.

Looking to answer the qualitative aspect of the OP's original question: What if I know I only want two principal components in order to visualize my data in 2-dimensions ? ICs can be rather helpful as ICA tries to minimize the mutual information among the projected data.

Check for example my following shamelessly ripped-off figure (from the excellent Bayesian Reasoning and Machine Learning by David Barber (Sect 21.6) - I used this figure for a talk, I seem to have misplaced ($\approx$ lost) my original MATLAB code, if I find it I will edit the question to append it):

enter image description here

Here we have a sample of two dimensional data-points (green points). The yellow lines are along the two major modes of variation in the sample defined by us during data generation. We treat them as the "true components/mode of variation". As you see PCA's first component is indeed very close to the "true" component A. PC1 has to be the mode of maximal variation in the data. The second principal component though is constrained to be orthogonal to the first one; a condition not found in the true components. Therefore if you try to use the PCs as an alternative axis system for your data you won't get the optimal representation in terms of explanatory power (assuming that is want you want to achieve when you visualize your data; you show the data so people get the idea of what is going on). So check also ICA! It might be helpful. :)

(Once more, Peter Flom's answer is the correct answer in terms of principal components, I am huge fan of PCA don't get me wrong, just it is not always the optimal solution. As ttnphns mentioned the definition of best changes things significantly.)

(And don't be tempted to immediately equate orthogonality with statistical independence; they are related but not same; for example see the 1984 fun little paper Linearly Independent, Orthogonal, and Uncorrelated Variables by Rodger et al.)

usεr11852
  • 33,608
  • 2
  • 75
  • 117
  • This is a nice answer, but I think you should explain what are "true components". Without it, the distinction between PCA and ICA remains mystical. – ttnphns Dec 21 '13 at 11:34
  • @ttnphns Thank you for the suggestion; I added few sentences to clarify the matter. – usεr11852 Dec 22 '13 at 02:14
  • A helpful explanation and that paper you suggest by Roger et. al is extremely useful for getting your head around the distinction! – n1k31t4 Dec 10 '15 at 01:16
14

The first two are the two best first two. The second one takes the first one into account.

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
  • 1
    I don't think so, because the second PC is the best way to keep the variance which haven't been capture by the first PC. But the first PC is not build in order to optimize its complementarity with the second PC. – Ophelie Dec 20 '13 at 13:39
  • 5
    It still works out. Someone else can supply the math, but it does work out. – Peter Flom Dec 20 '13 at 13:44
  • I'd like to have the demonstration, because in my mind it's like the Hierarchical clustering. When you go up in the dendrogram and you decide to keep 4 clusters, the 4 clusters are not the best way to divide the population in 4,because its merge clusters which have not been optimize for that merging – Ophelie Dec 20 '13 at 13:49
  • I am about to walk into a meeting, but I can put up the math this afternoon if someone hasn't already done so. In the mean time, I recommend checking out this book http://statweb.stanford.edu/~tibs/ElemStatLearn/ – Ellis Valentiner Dec 20 '13 at 13:54
  • 13
    +1. The math comes down to this: after diagonalizing the covariance matrix, it becomes evident that the total variance accounted for by any $k$-dimensional subspace cannot exceed the sum of the $k$ largest eigenvalues. Choosing the first $k$ principal components obviously achieves this upper bound, QED. – whuber Dec 20 '13 at 14:00
  • 1
    Ophelie, this graphical demonstration might help you understand what's going on in PCA http://stats.stackexchange.com/questions/76906/how-can-i-interpret-what-i-get-out-of-pca/76911#76911 – David Marx Dec 21 '13 at 09:54
  • 1
    This later thread covers the math and elaborates on whuber's comment above: [Why does PCA maximize variance of the projection?](http://stats.stackexchange.com/questions/102658) @Peter, perhaps you could even edit the link to this thread into your answer... – amoeba Jan 15 '15 at 11:30
-1

Please correct any errors you may find

Principal Components Analysis

Given data matrix $X \in \mathbb{R}^{\text{n x p}}$, where we have $n$ observations and $p$ column vectors, assume the columns of $X$ are centered with mean 0.

For any $v \in \mathbb{R}^p$, the vector $Xv \in \mathbb{R}^n$ has sample mean zero and sample variance $\frac{1}{n}(Xv)^T(Xv)$.

The first principal component direction $v_1 \in \mathbb{R}^p$ is

$v_1 = \underset{||v||_2=1}{\text{argmax}}~(Xv)^T(Xv)$

so the first principal component direction of $X$ is the unit vector $v_1 \in \mathbb{R}^p$ that maximizes the sample variance of $Xv_1 \in \mathbb{R}^p$.

The normalized first principal component score is

$\frac{(Xv_1)}{\sqrt{(Xv_1)^T(Xv_1)}}$

and the amount of variance explained by the first component is just

$\frac{\sqrt{(Xv_1)^T(Xv_1)}}{n}$ The second principal component direction is the unit vector with $v_2^Tv_1=0$, such that $Xv_2 \in \mathbb{R}^p$ has the maximum sample variance over all unit vectors orthogonal to $v_1$.

Now we can generalize further

$v_k = \underset{\underset{v^Tv_j=0, j=1,...k-1}{||v||_2=1}}{\text{argmax}}~(Xv)^T(Xv)$

for the $k^\text{th}$ principal component direction and score.

Ellis Valentiner
  • 1,063
  • 8
  • 21
  • 2
    This post seems to have stopped before it reached an answer to the question. Did the last part somehow get lost? – whuber Dec 20 '13 at 19:51