5

I'm looking to analyzing data from a study and previous studies that are similar have used either PCA or hierarchical linear regression to analyze the data. I've used both PCA and linear regression previously. From my understanding PCA breaks the data down into principal components and is useful for learning what factors may be strong indicators of our dependent variable, and that linear regression can be used to compare correlation.

How should I be approaching this? If I'm simply wanting to find out what correlates the strongest with my studies dependent variable what would be the best option? Can I use both PCA and then hierarchical linear regression?

Ben
  • 91,027
  • 3
  • 150
  • 376
4ntibody
  • 51
  • 1
  • 1
  • 2

4 Answers4

10

PCA does not involve a dependent variable: All the variables are treated the same. It is primarily dimension reduction method.

Factor analysis also doesn't involve a dependent variable, but its goal is somewhat different: It is to uncover latent factors.

Some people use either the components or the factors (or a subset of them) as independent variables in a later regression. This can be useful if you have a lot of IVs: If you want to reduce the number while losing as little variance as possible, that's PCA. If you think these IVs represent some factors, that's FA.

If you think there are factors, then it may be best to use FA; but if you are just trying to reduce the number of variables, then there is no guarantee that the components will relate well to the DV. Another method is partial least squares. That does include the DV.

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
5

These techniques are not exclusive, and they can be complimentary.

PCA is a dimension reduction technique. The number of dimensions in your dataset corresponds to the number of observations you have per case. For example, imagine your data is survey data, and you administered a 100 item questionnaire. Each individual who completed the questionnaire is represented by a single point in 100 dimensional space. The goal of PCA is to simplify this space in such a way that the distribution of points is preserved in fewer dimensions. This simplification can help you to describe the data more elegantly, but it can also reveal the dominant trends in your data. A great explanation of PCA can be found here: Making sense of principal component analysis, eigenvectors & eigenvalues

Hierarchical linear regression is used to determine whether a predictor (or set of predictors) explains variance in an outcome variable over and above some other predictor (or set of predictors). For example, you may want to know if exercising (IV1) or eating well (IV2) is a better predictor of cardiovascular health (DV). Hierarchical linear regression can help answer this question.

If your data is complex (i.e. you have many variables) you can apply PCA to reduce the number of variables/find the "latent variables". These latent variables can then be used in the hierarchical linear regression.

Best of luck!

unicoder
  • 76
  • 6
2

As other answers have said, PCA and Linear Regression (in general) are different tools.

PCA is an unsupervised method (only takes in data, no dependent variables) and Linear regression (in general) is a supervised learning method. If you have a dependent variable, a supervised method would be suited to your goals.

If you're trying to find out which variables in your data capture most of the variation in the data, PCA is a useful tool.

Alexander
  • 211
  • 1
  • 2
  • 6
1

If you are just looking for correlation between variables, you can estimate this simply with the correlation coefficient. It will tell you the strength of the correlation between two variables.

Juan
  • 11
  • 1