1

I want to know what elements are (varimax-)rotated when I rotate after PCA and after Factor Analysis.

Let’s assume a standardized data vector $X$ of dimension $N \times q$.

In PCA, I have the formula: $F=XL$

In Factor Analysis, I have: $X=FL^T+E$

F is of dim $N \times k$, and L is of dim $q \times k$.

I obtain the loadings for PCA by applying SVD on X and reading out one of the eigenvector matrices after having sorted them according to size of their eigenvalues. To make it clear, I decide to use the two components with the largest variance. Lets call them $C_{1}$ and $C_{2}$. So the eigenvectors from the SVD are equal to the loadings which are the new axes.

For Factor Analysis, I use for simplifying reasons „Principal Component Factors“ to extract my loadings because I do not have to estimate the specific variance - it arises as the residual variance after having calculated the factors (s. Rencher 2003 p.415-421). For PCF, I apply an SVD on $X$ and multiply the values in an eigenvector with the square root of their corresponding eigenvalues (so the main difference between the procedures at this point is only the multiplication of the eigenvectors with their eigenvalues – but still: it's Factor Analysis!). Again, I decide to retain the two factors with the largest eigenvalues. Lets call them $F_{1}$ and $F_{2}$.

I acknowledge that the factor analysis solution for the loadings is unique up to an orthogonal rotation.

Now on rotation. As far as I understand, I rotate orthogonally in PCA by multiplying the eigevectors $U_{1}$ and $U_{2}$ with an appropriate 2x2 matrix. So I rotate the axes (once more), but apparently I am still in space of the original variables, right? While my eigenvector) are still orthogonal to each other, my components (calculated as above =XL) are now correlated. I understand that this is drawback.

But what happens in factor analysis? My coordinate system is now spanned by the factors and not by the original variables, right? Somehow I lack intiution for I still haven’t estimated F.

Still, the procedure seems to be absolutely the same as before, because I rotate the loading matrix and the results seem very similar. They would be equal, I suppose, if the loadings in PCA would be multiplied by the square root of their eigenvalues, too. However, the drawback of correlated factors does not occur and I can safely rotate.

Having read @amoeba’s answer to Is PCA followed by a rotation (such as varimax) still PCA? , my question could not be answered maybe because I could not understand what the approach for FA was.

Community
  • 1
MaHo
  • 391
  • 2
  • 11
  • `eigenvectors from the SVD are equal to the loadings`; `so the main difference between the procedures is only the multiplication of the eigenvectors with their eigenvalues`. I stopped reading (excuse me please) upon encountering those lines. If, for you, that is the core distinction between PCA and FA then you must be missing a lot in your understanding. – ttnphns Mar 27 '15 at 15:47
  • Where did I say that? In this case, i.e. PCF as extraction method, it is the difference to obtain the loading matrix. I am very careful when it comes to differences between FA and PCA, in particular acknowleding that there is no such thing as FA, but many variants. – MaHo Mar 27 '15 at 17:45
  • After I wrote a long answer (that got zero upvotes) and @DStroet and me exchanged almost 20 comments trying to clarify this issue, he decided to close this question as a duplicate of the mentioned thread. I have now edited my answer there, trying to make it more clear and incorporating some parts of our discussion here. I don't see a point in keeping my answer in this thread, so I am deleting it (together with all the comments thread) to remove the clutter and to avoid potential confusion in the future. – amoeba Mar 31 '15 at 21:42

0 Answers0