14

As part of a University assignment, I have to conduct data pre-processing on a fairly huge, multivariate (>10) raw data set. I'm not a statistician in any sense of the word, so I'm a little confused as to what's going on. Apologies in advance for what is probably a laughably simple question - my head's spinning after looking at various answers and trying to wade through the stats-speak.

I've read that:

  • PCA allows me to reduce the dimensionality of my data
  • It does so by merging / removing attributes / dimensions that correlate a lot (and thus are a little unnecessary)
  • It does so by finding eigenvectors on covariance data (thanks to a nice tutorial I followed through to learn this)

Which is great.

However, I'm really struggling to see how I can apply this practically to my data. For instance (this is not the data set I'll be using, but an attempt at a decent example people can work with), if I were to have a data set with something like...

PersonID     Sex     Age Range    Hours Studied     Hours Spent on TV      Test Score     Coursework Score 
1            1       2            5                 7                      60             75
2            1       3            8                 2                      70             85 
3            2       2            6                 6                      50             77
...          ...     ...          ...               ...                    ...            ...

I'm not quite sure how I would interpret any results.

Most of the tutorials I've seen online seem to give me a very mathematical view of PCA. I've done some research into it and followed them through - but I'm still not entirely sure what this means for me, who's just trying to extract some form of meaning from this pile of data I have in front of me.

Simply performing PCA on my data (using a stats package) spits out an NxN matrix of numbers (where N is the number of original dimensions), which is entirely greek to me.

How can I do PCA and take what I get in a way I can then put into plain english in terms of the original dimensions?

Community
  • 1
nitsua
  • 243
  • 1
  • 2
  • 5
  • 3
    Your example data shows a mixture of data types: Sex is dichotomous, Age is ordinal, the other 3 are interval (and those being in different units). Doing linear PCA is right for interval data (but you have first to z-standardize those variables, because of the units). It is debatable whether PCA is appropriate for [binary or dichotomous](http://stats.stackexchange.com/q/16331/3277) data. You should not use ordinal data in linear PCA. But the main question with your example data: why _at all_ to do PCA with it; what sense could it make in this case? – ttnphns Nov 18 '13 at 18:17
  • (Please correct me if I'm wrong) I believe that PCA is/can be very useful for helping to find trends in the data and to figure out which attributes can relate to which (which I guess in the end would lead to figuring out patterns and the like). My assignment details that I have this massive data set and I just have to apply clustering and classifiers, and one of the steps it lists as vital to pre-processing is PCA. Would it help if I tried to extract some second order attributes from the data set I have to try and get them all in interval data? – nitsua Nov 18 '13 at 18:46
  • 3
    I only can recommend you, at present, to read more on PCA (on this site, too). Many uncertainties will surely go away. – ttnphns Nov 18 '13 at 18:54
  • This pdf is useful for the [application of PCA](http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf). The other posts here do a good job as describing the "meaning" of your PCA results. – ams Nov 18 '13 at 18:45
  • Many fine links above, here is a short example that "could" give you a good feel about PCA in terms of regression, with a practical example and very few, if at all, technical terms. http://sites.stat.psu.edu/~ajw13/stat505/fa06/16_princomp/10_princomp_reg_example.htm – leviathan Nov 18 '13 at 22:06
  • ttnphn's comment about the varying types of data in your set is important. PCA works best when the variables are of the same type. You might get credit for being aware of the risks of preprocessing the data in a way liable to produce poor results. – conjectures Nov 19 '13 at 17:43

3 Answers3

15

Pages 13-20 of the tutorial you posted provide a very intuitive geometric explanation of how PCA is used for dimensionality reduction.

The 13x13 matrix you mention is probably the "loading" or "rotation" matrix (I'm guessing your original data had 13 variables?) which can be interpreted in one of two (equivalent) ways:

  1. The (absolute values of the) columns of your loading matrix describe how much each variable proportionally "contributes" to each component.

  2. The rotation matrix rotates your data onto the basis defined by your rotation matrix. So if you have 2-D data and multiply your data by your rotation matrix, your new X-axis will be the first principal component and the new Y-axis will be the second principal component.

EDIT: This question gets asked a lot, so I'm just going to lay out a detailed visual explanation of what is going on when we use PCA for dimensionality reduction.

Consider a sample of 50 points generated from y=x + noise. The first principal component will lie along the line y=x and the second component will lie along the line y=-x, as shown below.

enter image description here

The aspect ratio messes it up a little, but take my word for it that the components are orthogonal. Applying PCA will rotate our data so the components become the x and y axes:

enter image description here

The data before the transformation are circles, the data after are crosses. In this particular example, the data wasn't rotated so much as it was flipped across the line y=-2x, but we could have just as easily inverted the y-axis to make this truly a rotation without loss of generality as described here.

The bulk of the variance, i.e. the information in the data, is spread along the first principal component (which is represented by the x-axis after we have transformed the data). There's a little variance along the second component (now the y-axis), but we can drop this component entirely without significant loss of information. So to collapse this from two dimensions into 1, we let the projection of the data onto the first principal component completely describe our data.

enter image description here

We can partially recover our original data by rotating (ok, projecting) it back onto the original axes.

enter image description here

The dark blue points are the "recovered" data, whereas the empty points are the original data. As you can see, we have lost some of the information from the original data, specifically the variance in the direction of the second principal component. But for many purposes, this compressed description (using the projection along the first principal component) may suit our needs.

Here's the code I used to generate this example in case you want to replicate it yourself. If you reduce the variance of the noise component on the second line, the amount of data lost by the PCA transformation will decrease as well because the data will converge onto the first principal component:

set.seed(123)
y2 = x + rnorm(n,0,.2)
mydata = cbind(x,y2)
m2 = colMeans(mydata)

p2 = prcomp(mydata, center=F, scale=F)
reduced2= cbind(p2$x[,1], rep(0, nrow(p2$x)))
recovered = reduced2 %*% p2$rotation

plot(mydata, xlim=c(-1.5,1.5), ylim=c(-1.5,1.5), main='Data with principal component vectors')
arrows(x0=m2[1], y0=m2[2]
       ,x1=m2[1]+abs(p2$rotation[1,1])
       ,y1=m2[2]+abs(p2$rotation[2,1])
       , col='red')
arrows(x0=m2[1], y0=m2[2]
       ,x1=m2[1]+p2$rotation[1,2]
       ,y1=m2[2]+p2$rotation[2,2]
       , col='blue')

plot(mydata, xlim=c(-1.5,1.5), ylim=c(-1.5,1.5), main='Data after PCA transformation')
points(p2$x, col='black', pch=3)
arrows(x0=m2[1], y0=m2[2]
       ,x1=m2[1]+abs(p2$rotation[1,1])
       ,y1=m2[2]+abs(p2$rotation[2,1])
       , col='red')
arrows(x0=m2[1], y0=m2[2]
       ,x1=m2[1]+p2$rotation[1,2]
       ,y1=m2[2]+p2$rotation[2,2]
       , col='blue')
arrows(x0=mean(p2$x[,1])
      ,y0=0
      ,x1=mean(p2$x[,1])
      ,y1=1
      ,col='blue'
       )
arrows(x0=mean(p2$x[,1])
       ,y0=0
       ,x1=-1.5
       ,y1=0
       ,col='red'
)
lines(x=c(-1,1), y=c(2,-2), lty=2)


plot(p2$x, xlim=c(-1.5,1.5), ylim=c(-1.5,1.5), main='PCA dimensionality reduction')
points(reduced2, pch=20, col="blue")
for(i in 1:n){
  lines(rbind(reduced2[i,], p2$x[i,]), col='blue')
}

plot(mydata, xlim=c(-1.5,1.5), ylim=c(-1.5,1.5), main='Lossy data recovery after PCA transformation')
arrows(x0=m2[1], y0=m2[2]
       ,x1=m2[1]+abs(p2$rotation[1,1])
       ,y1=m2[2]+abs(p2$rotation[2,1])
       , col='red')
arrows(x0=m2[1], y0=m2[2]
       ,x1=m2[1]+p2$rotation[1,2]
       ,y1=m2[2]+p2$rotation[2,2]
       , col='blue')
for(i in 1:n){
  lines(rbind(recovered[i,], mydata[i,]), col='blue')
}
points(recovered, col='blue', pch=20)
Community
  • 1
David Marx
  • 6,647
  • 1
  • 25
  • 43
  • 5
    David, please, refrain from use terms "rotation matrix" (aka eigenvectors) and "loading matrix" interchangeably. For purity and not to mislead people. Loadings in PCA are eigenvectors _after_ normalizing by their eigenvalues. Thank you. – ttnphns Nov 18 '13 at 18:29
  • 3
    @ttphns I think it completely depends on what package you use. Often these terms are completely interchangeable. Consider the usage of "loadings" here: http://www.utdallas.edu/~herve/abdi-awPCA2010.pdf. Also reference the R function `prcomp` where the loadings matrix is just the matrix whose columns are unit eigenvectors. I think you are being unnecessarily technical, and in most cases I believe these terms are used interchangeably. – David Marx Nov 18 '13 at 18:37
  • 4
    Sorry, but I would disagree. Reason: remember that loadings are both meaningful (and in the same sense!) to PCA and factor analysis. Loadings are directly comparable to the correlations/covariances. Eigenvectors are the rotation cosines. In factor analysis, many methods do not deal with rotation (_at_ extraction phase). So, mixing the two terms are indeed misleading to students and should be avoided. – ttnphns Nov 18 '13 at 18:50
  • Smaller point: correct spelling is always and only "principal", not "principle". I've edited accordingly, but one image I can't edit. – Nick Cox Nov 20 '13 at 13:34
  • 1
    Another visual can be found here http://setosa.io/ev/principal-component-analysis/ – SIslam Apr 23 '16 at 17:35
2

I would say your question is a qualified question not only in cross validated but also in stack overflow, where you will be told how to implement dimension reduction in R(..etc.) to effectively help you identify which column/variable contribute the better to the variance of the whole dataset.

The PCA(Principal Component Analysis) has the same functionality as SVD(Singular Value Decomposition), and they are actually the exact same process after applying scale/the z-transformation to the dataset.

Here are some resources that you can go through in half an hour to get much better understanding.

I am not capable to give a vivid coding solution to help you understand how to implement svd and what each component does, but people are awesome, here are some very informative posts that I used to catch up with the application side of SVD even if I know how to hand calculate a 3by3 SVD problem.. :)

  1. Coursera Data Analysis Class by Jeff Leek: Video Lecture / Class Notes
  2. A Very Informative student post
  3. A post from American Mathematical Society.
B.Mr.W.
  • 151
  • 1
  • 10
  • Jeff Leek's class is very good for getting a feeling of what you can do with PCA. The good thing is that it does not get into complex mathematical/statistical details (which can be found in plenty of other places) but rather provides an hands-on approach showing how to really use it on data. – nico Nov 19 '13 at 21:23
1

In PCA you want to describe the data in fewer variables. You can get the same information in fewer variables than with all the variables. For example, hours studied and test score might be correlated and we do not have to include both.

In your example, let's say your objective is to measure how "good" a student/person is. Looking at all these variables, it can be confusing to see how to do this. PCA allows us to clearly see which students are good/bad.

If the first principal component explains most of the variation of the data, then this is all we need. You would find the correlation between this component and all the variables. "Large" correlations signify important variables. For example, the first component might be strongly correlated with hours studied and test score. So high values of the first component indicate high values of study time and test score.

pcaguy
  • 11
  • 2