2

After reading this, I am still not sure how to intuitively understand why eigenvectors of a covariance matrix represents direction of the LARGEST variance.

Need more explanation on:

Then $w^{T}Cw$ simplifies to $\sum\lambda_i w_i^{2}$ , in other words the variance is given by the weighted sum of the eigenvalues. It is almost immediate (why?) that to maximize this expression one should simply take $w=(1,0,0,…,0)$, i.e. the first eigenvector, yielding variance $\lambda_1$."

Here $C$ should be the original $C$ before decomposition? or both $C$ and $w$ have changed basis? If they changed basis, how do we know the original variance $w^{T}Cw$ is the same as the new variance $w^{T}Cw$?

Community
  • 1
iwbabn
  • 291
  • 1
  • 2
  • 9
  • There are two things going on. First, let's imagine that $C$ is diagonal (and with ordered values on the diagonal) to begin with. Is it then clear why $w=(1,0,0,...0)$ maximizes variance? This refers to your "why" in the middle of my quote. – amoeba Feb 01 '17 at 07:52
  • Second, what happens if $C$ is not diagonal. This quote refers to $C$ already diagonalized, i.e. in the changed basis. How do we know the variance did not change? Well, the basis was changed by pure rotation. So in 2D you can imagine everything drawn on the piece of paper and you are rotating this piece of paper. Isn't it intuitively obvious that variances will not change because of that? – amoeba Feb 01 '17 at 07:54
  • Thank you. I think it would be helpful as well to add that orthogonal matrix preserves length: [link](http://math.stackexchange.com/questions/612936/why-are-orthogonal-matrices-generalizations-of-rotations-and-reflections) – iwbabn Feb 01 '17 at 15:16
  • So is it all clear now? – amoeba Feb 01 '17 at 15:17
  • I inserted some additional explanations about these two points in my linked answer. Can you take a look and tell me if you think that part in that answer became clear now? If you think yes, then we can close this question as a duplicate of that one (I already vote now). – amoeba Feb 02 '17 at 09:11

0 Answers0