1

It is well known that the solution to the optimization problems proposed in Principal Components and Canonical Correlation Analysis are given by the solution to eigenvalue problems and generalized eigenvalue problems respectively. My linear algebra is quite rusty and I wanted to know if anyone could give me a semi-formal, albeit intuitive explanation on why this is so.

Nick Cox
  • 48,377
  • 8
  • 110
  • 156
JEquihua
  • 3,525
  • 2
  • 24
  • 44
  • 2
    Step-by-step math algorithm of CCA is given [here](http://stats.stackexchange.com/a/77309/3277). Algorithm of PCA is easier and is repeated in numerous places on this site; to cite some of my own posts: [decomposition](http://stats.stackexchange.com/q/79043/3277); [loadings](http://stats.stackexchange.com/a/17102/3277); [loadings vs eigenvectors](http://stats.stackexchange.com/a/35653/3277); [complete output](http://stats.stackexchange.com/a/80576/3277). – ttnphns Mar 03 '14 at 06:45
  • The problem of CCA is a bit closer to multivariate multiple regression than to PCA: [1](http://stats.stackexchange.com/a/65817/3277), [2](http://stats.stackexchange.com/a/31468/3277). – ttnphns Mar 03 '14 at 06:46
  • If your question is specifically about eigen-decompositions... In PCA, we have one set of variables, and so the matrix to decompose is symmetric (=> standard eigen-problem). In CCA or discriminant analysis we have 2 sets counter each other, so the matrix is asymmetric (=> generalized eigen-problem, it can be solved in various bypassing tricks). – ttnphns Mar 03 '14 at 06:57
  • 3
    I recommend this simple and [concise paper](http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.diva-portal.org%2Fsmash%2Fget%2Fdiva2%3A288565%2FFULLTEXT01.pdf&ei=nFQUU66qF7OM7Aanh4DoDA&usg=AFQjCNHvHTvq7t29fv9cIVW_0qLsLWNzig&sig2=pGvoqXagrz4Af3ExKCDaTQ&bvm=bv.61965928,d.ZGU): M. Borga, T. Landelius, H. Knutsson (1998). A Unified Approach to PCA, PLS, MLR and CCA. – user603 Mar 03 '14 at 10:11
  • @user603 I liked the paper very much. Why does it say that the Rayleigh Quotient is an energy function? what is that? Could you help me understand that terminology? Thank you! – JEquihua Mar 03 '14 at 16:28
  • @JEquihua: You're welcome. I've often seen (electrical) engineers refer to loss functions as 'energy functions'. I don't know if this is also the sense in which the expression is used in this paper (but it seems to make sense). – user603 Mar 03 '14 at 16:33
  • I think it merely refers to an objective function. Not sure yet. If you just describe the paper: PCA, PLS, CCA, MLR are special cases of generalized eigenproblems where you find subspaces of maximum variance, maximum covariance, maximum correlation, minimum squared error respectively and... etc I would accept the answer. Thank you very much. – JEquihua Mar 03 '14 at 16:40
  • 1
    Perhaps my CCA by rotation is instructive for an old rusty linear-algebra dog... See http://go.helms-net.de/stat/sse/cancorr150712.htm – Gottfried Helms Jul 13 '15 at 16:50

0 Answers0