Why are PCA eigenvectors orthogonal but correlated?

Question

I've seen some great posts explaining PCA and why under this approach the eigenvectors of a (symmetric) correlation matrix are orthogonal. I also understand the ways to show that such vectors are orthogonal to each other (e.g. taking the cross-products of the matrix of these eigenvectors will result in a matrix with off-diagonal entries that are zero).

My first question is, when you look at the correlations of a PCA's eigenvectors, why are the off-diagonal entries of the correlation matrix non-zero (i.e. how can the eigenvectors be correlated if they are orthogonal)?

This question is not directly about PCA, but I put it in this context since that is how I ran into the issue. I am using R and specifically the psych package to run PCA.

If it helps to have an example, this post on StackOverflow has one that is very convenient and related (also in R). In this post, the author of the best answer shows that the PCA loadings (eigenvectors) are orthogonal by using Factor Congruence or cross-products. In his example, the matrix L is the PCA loadings matrix. The only thing that is not on this link is that cor(L) will produce the output I am asking about showing the non-zero correlations between the eigenvectors.

I am especially confused about how orthogonal vectors can be correlated after reading this post, which seems to prove that orthogonality is equivalent to lack of correlation: Why are PCA eigenvectors orthogonal and what is the relation to the PCA scores being uncorrelated?

My second question is: when the PCA eigenvectors are used to calculate PCA scores, the scores themselves are uncorrelated (as I expected)... is there a connection to my first question about this, why eigenvectors are correlated but not the scores?

Answer 1

Let $X$ be a random vector $X=(x_1,x_2,\cdots,x_d)^T $ with expected value $\mu$ and variance $\Sigma$. We are looking for such ordered vectors $u_i$, that maximize the variance of $u_i^TX$. Essentialy we are solving $$\max\limits_{u_i} Var(u_i^TX)$$ $$s.t. \quad u_i^T u_i=1.$$ Because we are only interested in the direction of such vectors, we are additionally assuming the unit length of vectors $u_i^T u_i=1$. Vectors $u_i$ are actually not random (because we are working theoretically now, in reality we are replacing the unknown $\Sigma$ and unknown $\mu$ with Empirical sample covariance matrix and mean respectively, @whuber was explaining this from a different perspective) so $$Var(u_i^TX)=u_i^T\Sigma u_i.$$ The optimization problem can be trivially solved by using the Lagrange function $$L(u_i,\lambda_i):=u_i^T \Sigma u_i -\lambda_i(u_i^Tu_i-1).$$ From there we get the necessary condition for constrained extrema $$ \frac{\partial L(u_i,\lambda_i)}{\partial u_i} = 2\Sigma u_i -2\lambda_i u_i=0,$$ which can be reduced to $$\Sigma u_i =\lambda_i u_i,$$ that is by definition the problem of eigenvalues and eigenvectors. Because $\Sigma$ is symmetric and positive semidefinite matrix, the spectral theorem applies and we are able to find orthonormal basis that satisfies $\Sigma=Q\Lambda Q^{-1}=Q\Lambda Q^T$, where $Q$ is made of orthogonal eigenvectors and $\Lambda$ is a diagonal matrix with eigenvalues which are all real.

Now we can show that $$cov(u_i^TX,u_j^TX)=u_i^T\Sigma u_j=\lambda_j u_i^Tu_j=0, \quad \forall j \neq i.$$ Trivially for $i=j: \quad cov(u_i^TX,u_j^TX)=\lambda_i.$ So not the eigenvectors, but the projections are uncorrelated.

Answer 2

Note that L is the loadings matrix, aka the eigenvectors themselves. This isn't the PCA data matrix. The eigenvectors are bound to provide orthogonality but not $cov=0$. For example, take the matrix:

> X <- iris
> X$Species <- as.numeric(X$Species)
> head(X)
  Sepal.Length Sepal.Width Petal.Length Petal.Width Species
1          5.1         3.5          1.4         0.2       1
2          4.9         3.0          1.4         0.2       1
3          4.7         3.2          1.3         0.2       1
4          4.6         3.1          1.5         0.2       1
5          5.0         3.6          1.4         0.2       1
6          5.4         3.9          1.7         0.4       1

In PCA, not only you get the eigenvectors of the covariance/correlation matrix (depends on the method) but they are also being orthonormal (that is, $\left \| u_j \right \|=1$ for each eigenvector $u_j$), so we get:

> prcomp(X)$rotation
                     PC1         PC2        PC3         PC4        PC5
Sepal.Length  0.33402494 -0.68852577  0.4414776 -0.43312829  0.1784853
Sepal.Width  -0.08034626 -0.68474905 -0.6114140  0.30348725 -0.2423462
Petal.Length  0.80059273  0.09713877  0.1466787  0.49080356 -0.2953177
Petal.Width   0.33657862  0.06894557 -0.4202025  0.06667133  0.8372253
Species       0.35740442  0.20703034 -0.4828930 -0.68917499 -0.3482135

and

> cor(prcomp(X)$rotation)
            PC1         PC2         PC3          PC4          PC5
PC1  1.00000000  0.62712979  0.57079328  0.147574029 -0.072934736
PC2  0.62712979  1.00000000 -0.22763304 -0.058852698  0.029086459
PC3  0.57079328 -0.22763304  1.00000000 -0.053565825  0.026473556
PC4  0.14757403 -0.05885270 -0.05356582  1.000000000  0.006844526
PC5 -0.07293474  0.02908646  0.02647356  0.006844526  1.000000000

but note that the PCA'd data is

> head(prcomp(X)$x)
           PC1        PC2          PC3          PC4          PC5
[1,] -2.865415 -0.2962946 -0.041870662 -0.078464301 -0.032047052
[2,] -2.892047  0.1837851  0.175540800 -0.143582265  0.053428970
[3,] -3.054980  0.1748266 -0.049705391 -0.045339514 -0.001205543
[4,] -2.920230  0.3315818 -0.003376012  0.065785303 -0.053882996
[5,] -2.906852 -0.2959169 -0.147159821 -0.004802747 -0.074130194
[6,] -2.489852 -0.7338212 -0.194029844  0.073567444  0.003409809

and its correlation is

> round(cor(prcomp(X)$x),14)
    PC1 PC2 PC3 PC4 PC5
PC1   1   0   0   0   0
PC2   0   1   0   0   0
PC3   0   0   1   0   0
PC4   0   0   0   1   0
PC5   0   0   0   0   1

Answer 3

A simpler example of how this isn't a contradiction:

Suppose $A$ and $B$ are random vectors in $\mathbb{R}^2$ whose distribution is given by:

$$ \mathbb{P}\left(A = \begin{bmatrix} +1 \cr 0 \end{bmatrix}, \:B = \begin{bmatrix} 0 \cr +1 \end{bmatrix}\right) = \frac12\\ \mathbb{P}\left(A = \begin{bmatrix} 0 \cr +1 \end{bmatrix}, \:B = \begin{bmatrix} -1 \cr 0 \end{bmatrix} \right) = \frac12\\ $$

So $B$ is always equal to $A$ rotated by 90 degrees anticlockwise. Then $A$ and $B$ are always orthogonal, but they are not uncorrelated.