1

i) What is the main role of "only" trying to find orthogonal components in PCA?

I can understand, that we would not want a zero-solution as well as find directions that are orthogonal in order to explain most of the variance. When we look at the problem in terms of finding a projection matrix that preserves the gram matrix of mean centered data,

ii) What would an optimization under a non orthonormal constraint produce, as long as we make sure the solution is non-zero? Why would it be or not be useful?

I am aware of the concept of non-orthogonal eigen-functions as well, just in case you want to generalize the answer to a Kernel PCA, that would be fine as well.

hearse
  • 2,355
  • 1
  • 17
  • 30

1 Answers1

4

i) PCA only looks for orthogonal components because it is computationally easiest. Linear algebra already had convenient computational solutions for finding orthogonal basis vectors (the eigenvalue problem) so PCA copped it. There may very well be "better" basis vectors that are not orthogonal, but limiting the search to only orthogonal vectors severely limits the subspace (and also makes computation easier). For instance, page 9 of A Tutorial on Principal Component Analysis goes into detail on how PCA fails miserably to deduce the "right" basis for the movement of a ferris wheel, as there is no orthogonal basis to describe its motion. tracking a person on a ferris wheel

ii) Well, the easiest thing we could say about these new solutions is they would not be orthogonal! If your data set is not orthogonal, such as the ferris wheel example, then PCA would not be able to find these solutions.

Oren Milman
  • 1,132
  • 11
  • 25
tkg
  • 72
  • 4
  • is there another link for that paper, or a nice snippet to insert into the answer? – bright-star May 12 '14 at 08:12
  • -1. I don't really understand what the computational easiness has to do with it. The point is, if one takes orthogonality constraint out, then one should put some other constraint in, otherwise you will get the first PC over and over again. – amoeba Jan 22 '15 at 17:34
  • The answer needs to have some further clarification. Why it is easier to calculate? Is it the only reason? – Catbuilts Oct 13 '20 at 19:05