0

I have been using PCA results and variable loadings on the factors generated to select original uncorrelated input variables for model making

I was putting what i thought were rotated factors from PCA into models and seeing what factors were significant. Then i would go back and see if i could make sense of the results in regards to the original input variables.

For example if rotated PC #4 was highly significant in my model and had a single loaded input variable (say variable 8 loaded onto PC #4 to 0.9) i would substitute PC#4 for variable #8.

This did not work for one data set and i found that the factors i was using were generated raw unrotated PCA factors. These are obviously much less distinguishable in terms of the original x variables.

Im not very familiar with the pure mathematical side of PCA but i had assumed the factors i generated related to the rotated factors not the unrotated factors.

The initial data set i worked on appeared to be successful as the unrotated and rotated factors were fairly similar thus i didnt come across the issue

Can i use rotated factors after PCA to make a model and then substitute these factors back to original X variables?

Samuel
  • 121
  • 10
  • *"I had assumed the PCs I generated related to the rotated factors, not the unrotated factors"* -- I am afraid you were wrong. PCA does not care about rotating, it cares only about maximizing variance of each component and any rotation would only interfere with this goal. You can first do the PCA and then rotate your PCs, but if you just do PCA and nothing else -- there is no reason to expect any "simple structure" in your PCs. – amoeba Mar 03 '14 at 14:10
  • I was hoping to use the rotated PCA factors in the way i have been using the unrotated (PCs).. i thought this was the idea of PCA overall – Samuel Mar 03 '14 at 14:19
  • I guess I am not exactly sure what you are asking. Note that are no question marks in your question. My point was simply that PCA does not rotate anything. Maybe you already know that. Do you do PCA and then, in addition, rotate the components? How? And what is your question? – amoeba Mar 03 '14 at 14:23
  • The software i use generates the PCs, rotates them (choice of rotation methods) and i thought then used these rotated factors as new variables. What i really want to do was use the rotated factors as a set of variables in a model so i can substitute them for original input variables. I guess my question is.. Can i use rotated factors after PCA to make a model and then substitute these factors back to original X variables? – Samuel Mar 03 '14 at 14:27
  • Well, yes, technically you can try, but how well this would work depends on what you are actually trying to achieve. I am also not sure how your last comment relates to your original question. E.g. you wrote: "I found that my PCs ... appear to be unrotated factors" -- do you mean your PCs after your software rotated them? So your problem is that the rotation is not very successful? Or do you mean that after you found that, you started rotating? In any case I would strongly advice you to edit your question and make it clearer. – amoeba Mar 03 '14 at 14:34
  • The software performed a rotation but saved the unrotated factors with no option to choose between the two. Seeing as the final output was rotated factors i assumed it had saved these (This is very much GUI driven software) – Samuel Mar 03 '14 at 14:52

2 Answers2

1

Just to clarify some language: the principal components don't get rotated: they are the rotation. When you transform your data using PCA, if you keep all the principal components, the new transformed data is equivalent to a rotation onto the basis defined by the principal components.

So, can you fit your model in the rotated basis? Sure. In fact, this is actually one method used to deal with multicollinearity, since the components of the new basis are necessarily going to be orthogonal.

Now, all that said: it's still not clear to me what you are trying to do with PCA, so I'm not sure if this answers your question.

Community
  • 1
David Marx
  • 6,647
  • 1
  • 25
  • 43
  • I think what OP means by "rotating" is rotation in the sense of factor analysis, which [can be done after PCA as well](http://stats.stackexchange.com/questions/2742/on-the-use-of-oblique-rotation-after-pca). – amoeba Mar 03 '14 at 17:43
  • Rotations (such as varimax) are done for interpretability of the components. If this is the OP's goal, they consider PCA as factor analysis (a stance that has its rights). But since, as David said, PCA is rotation itself, what a surprise if it happens another rotation brings back their initial data? :-) – ttnphns Mar 03 '14 at 18:04
  • Yes i need to perform a further rotation after PCA (varimax). If this is not done i cannot really relate my factors back to original variables. Really i am trying to reduce the number of original variables and at the same time reduce colinearity. – Samuel Mar 04 '14 at 09:05
0

In some cases, you can use your "rotated" predictor matrix to make the predictions. I assume the rotated matrix means score matrix defined in this wiki page: http://en.wikipedia.org/wiki/Principal_component_analysis

However, this is not PCA is for. Using the rotated matrix (i.e. score matrix) won't give you better result. The purpose of performing PCA is to reduce the dimensions of your variables. You have to look at your principal component variances (i.e. latent) to decide how many dimensions you would like to keep, then "rotate" the score matrix with reduced dimension back to your original predictor space.

First thing I suggest you to check is what is really the output of your software, is it a score matrix? or weights? as defined in the wiki page above. Then you can perform a dimension reduction to get a better result.

Does that answer you question?

Nick Cox
  • 48,377
  • 8
  • 110
  • 156
photunix
  • 9
  • 1
  • 2
    "The purpose of performing PCA is to reduce the dimensions of your variables." No, this is just one way PCA is used. This is not the only thing that PCA is good for. – David Marx Mar 03 '14 at 17:31