10

How would I use the output of a principal components analysis (PCA) in a generalized linear model (GLM), assuming the PCA is used for variable selection for the GLM?

Clarification: I want to use PCA to avoid using correlated variables in the GLM. However, PCA gives me output like .2*variable1+.5*variable3 etc. I'm used to just putting variables 1 and 3 into the GLM. So since PCA gives me a linear combination, should I for example put that into the GLM as a new variable (to account for systematic variation in the response by variables 1 and 3)?

amoeba
  • 93,463
  • 28
  • 275
  • 317
ciel
  • 123
  • 1
  • 1
  • 4

3 Answers3

13

It is possible and sometimes appropriate to use a subset of the principal components as explanatory variables in a linear model rather than the the original variables. The resulting coefficients then need to be be back-transformed to apply to the original variables. The results are biased but may be superior to more straightforward techniques.

PCA delivers a set of principal components that are linear combinations of the original variables. If you have $k$ original variables you still have $k$ principal components in the end, but they have been rotated through $k$-dimensional space so they are orthogonal to (ie uncorrelated with) eachother (this is easiest to think through with just two variables).

The trick to using PCA results in a linear model is that you make a decision to eliminate a certain number of the principal components. This decision is based on similar criteria to the "usual" black-art variable selection processes for building models.

The method is used to deal with multi-collinearity. It is reasonably common in linear regression with a Normal response and identity link function from the linear predictor to the response; but less common with a generalized linear model. There is at least one article on the issues on the web.

I'm not aware of any user-friendly software implementations. It would be fairly straightforward to do the PCA and use the resulting principal components as your explanatory variables in a generalized linear model; and then to translate back to the original scale. Estimating the distribution (variance, bias and shape) of your estimators having done this would be tricky however; the standard output from your generalized linear model will be wrong because it assumes you are dealing with original observations. You could build a bootstrap around the whole procedure (PCA and glm combined), which would be feasible in either R or SAS.

Peter Ellis
  • 16,522
  • 1
  • 44
  • 82
6

My answer is not for the original question, but comments on your approach.

First apply PCA, then run generalized linear model is not recommended. The reason is PCA will select variable importance by "variable variance" but not "how variable is correlated with the prediction target". In other words, the "variable select" can be totally misleading, that select not important variables.

Here is an example: left future shows x1 is the important to classify two types of points. But PCA shows opposite.

enter image description here

Details can be found in my answer here. How to decide between PCA and logistic regression?

Haitao Du
  • 32,885
  • 17
  • 118
  • 213
4

I would suggest you take a look at this paper. It does a nice job showing the relationship between gaussian family distributions and PCA-like learner systems.

http://papers.nips.cc/paper/2078-a-generalization-of-principal-components-analysis-to-the-exponential-family.pdf

EDIT

Synopsis : while many think of PCA from the geometric interpretation of finding the orthogonal vectors within a dataset most responsible for the variance and then providing parameters to correctly re-orient one's space to those vectors, this paper builds up PCA using exponential probability functions in the context of generalized linear models, and offers a more powerful extension of PCA for other probability functions within the exponential family. In addition, they build a PCA-like learner algorithm using bregman divergences. It's fairly easy to follow and for you, it seems like it could help you understand the link between PCA and generalized linear models.

citation :

Collins, Michael et al. "A Generalization of Principal Component Analysis to the Exponential Family". Neural Information Processing Systems

themantalope
  • 141
  • 4