6

I'm learning PCA in R language. I met two problems right now that I don't understand.

  1. I am performing a PCA analysis in R on a 318×17 dataset using some custom code. I take eigen function in R to find eigenvalues and eigenvectors. But my 1st and 3rd eigenvectors are of the opposite sign to my handbook. My second eigenvectors is almost the same.

  2. I know that given a square matrix A, the condition that characterizes an eigenvalue, $\lambda$, is the existence of a nonzero vector $x$ such that $Ax=\lambda x$; this equation can be rewritten as follows: $(A - \lambda)x=0$.

    Now I calculate covariance of my data and have eigenvalues. I want to solve this linear combination equation to find $x$ and compare with initial eigenvectors. When I take solve function in R, my $x$ vector is always zero.

Here are my questions: Why the sign is different? How to use solve function in R to find a non-zero vector $x$?

Community
  • 1
Kroll DU
  • 73
  • 1
  • 3
  • Regarding the first part of your question, I encountered the same wrinkle when working on this [post](http://stats.stackexchange.com/a/150978/67822). It came to no surprise that I could have followed through the rest of the derivation with the signs reversed. I don't remember exactly, but it could have amounted to a upper-right quadrant versus lower-left quadrant graphical plotting of the vectors with essentially the same meaning. – Antoni Parellada May 30 '15 at 13:35

1 Answers1

12

1) The definition of eigenvector $Ax = \lambda x$ is ambidextrous. If $x$ is an eigenvector, so is $-x$, for then

$$A(-x) = -Ax = -\lambda x = \lambda (-x)$$

So the definition of an eigenbasis is ambiguous of sign.

2) It's hard to know for sure, but I have a strong suspicion of what is happening here. Your equation

$$ (A - \lambda)x = 0 $$

is technically incorrect. The correct equation is

$$ (A - \lambda I)x$$

The first equation is often used as a shorthand for the second. In general, this is unambiguous, because there is no real mathematical way to subtract a vector from a square matrix, but it is abuse of notation. In R though, you have broadcasting. So if you do

> M <- matrix(c(1, 1, 1, 1), nrow=2)
> M - .5
     [,1] [,2]
[1,]  0.5  0.5
[2,]  0.5  0.5

its not really what you want. The proper way would be

> M - diag(.5, 2)
     [,1] [,2]
[1,]  0.5  1.0
[2,]  1.0  0.5

The reason you are getting zero solutions is that the matrix you are starting with $A$ is invertible. More than likely (almost surely), the matrix you get by subtracting the same number from every entry will also be invertible. For invertible matrices, the only solution to $Ax = 0$ is the zero vector.

Matthew Drury
  • 33,314
  • 2
  • 101
  • 132
  • Thanks @ Matthew Drry . You're right my equation is used as a shorthand for the second. It's true that if A is invertible, Ax=0 can be written as x=0*A(-1).So x is always zero vector. As I mentioned,the condition that characterizes an eigenvalue, λ, is the existence of a nonzero vector x such that Ax=λx. How R can find a nonzero vector x?Because when i took eigen function, I got nonzero eigenvectors.Does it mean that behind eigen function, R find eigenvectors by another way which is not the solve function? – Kroll DU May 31 '15 at 10:14
  • 2
    @KrollDU You won't be able to use solve to recover the eigenvectors, because `solve` is designed for *non-singular* systems, and the eigenvector system is, by design, singular. So no, R does not use `solve` internally in it's `eigen` computation. You can check this by looking at the source code for `eigen`, where you'll find `z – Matthew Drury May 31 '15 at 14:41
  • Looks like `La_rs` is a C function calling into a fortran routine, you can find it here: https://github.com/wch/r-source/blob/4adf54a3a9d92993f1d827396d7eb6cf6b60dbf7/src/modules/lapack/Lapack.c – Matthew Drury May 31 '15 at 14:42
  • 1
    @ Matthew Drury, You're right. Solve function can't be used to recover the eigenvectors. Thanks so much for your help. – Kroll DU May 31 '15 at 14:57
  • @ Mathew if the definition of an eigenbasis is ambiguous of sign, what's the impact to the final result? – Kroll DU May 31 '15 at 15:01
  • @KrollDU Could you clarify that last question? What do you mean by *impact to the final result*? – Matthew Drury May 31 '15 at 15:01