I can't seem to understand how PCA works. My (lack of) math knowledge don't help me either. I have read that the new set of variables has to be a linear combination of the old set.
What does that mean exactly? That there should be a way to multiply numbers a, b, c, ... with dimensions/variables x, y, z... and get the old bigger(!) set?
Can you answer my example: if I have 8 variables/dimension can they be reduced to 3? Or a vector of 3 components is not a linear combination of 8 and thus no?
Ion, PCA is just a specific case of orthogonal rotation. Let X be your n x p data matrix of n points in p dimensions (axes). To obtain this same cloud of points in a new set of axes somehow rotated in space relatively the old ones, you multiply X by a p x p matrix Q of cosines between the old axes (rows) and new axes (columns): $\bf{XQ=C}$ [1], where C is your new (rotated) coordinates. This formula says that each new dimension is a linear combination of p old dimensions. It also follows that $\bf {X=CQ^{-1}}$ or, since the rotation was orthogonal - $\bf Q$ is orthonormal matrix, - $\bf {X=CQ'}$ [2] which says that each old dimension is a linear combination of p new dimensions.
Now, PCA is virtually this rotation; what makes PCA special is that Q is not an arbitrary rotation matrix; it is the matrix of such a rotation so that the sum-of-squares (or variance, if your data had been centered) in the 1st column of C becomes maximal possible: that is, variability along the 1st principal component is maximized. Then, sum-of-squares of in the 2nd C column (2nd principal component) is second maximal. Etc. Each next component is a new axis which takes off less and less of multidimensional variability in the cloud. Hence, lion's share of the variability is accounted for by only few m (m<p) new axes (principal components).
In PCA, Q is called matrix of eigenvectors (these being its columns). If you retain just m first components, by retaining just m first columns in Q, you still can use formula [1] to obtain component scores for the m components - the points' coordinates on these m dimensions. So, whatever is m, each component remains a linear combination of original variables. However, using then formula [2] to obtain p original variables from m components won't give you original variables exactly: each original variable will be a linear combination of m components plus some error term. If you perform linear regression (without constant term) of each original variable by m components as predictors you will see that regression coefficients you get are the elements of Q.
+1 for ttnphns, but I'll try to give tl;dr, math free version.
Your doubts are fully justified -- one cannot stuff 8 dims as <8 linear combinations in a general case. What PCA really does is that it converts 8 dims into 8 linear combinations in such a way that it stuffs how much diversity of the data possible to the first dim, than stuffs the most of remains in the second one and so on -- thus one may expect that the last dims contain only noise coming from errors and noises in the original data and may be omitted, what leads to a reduction of dimensionality.
This way one can imagine it as lossy compression algorithm like MP3 or JPEG -- it dumps some of the original information, but hopefully only this that doesn't matter.
Here's my stab at this; completely without math, just some basic principles and a picture. You asked for it. ;)
Consider the scenario in the picture below. You have 2D data points along the X and Y axis. You could use the PCA to find the principal axis P.
The point of this analysis is that if your data are distributed this way, you don't really need both X and Y to work with them. You might as well only use one dimension, along P.
If you have N-dimensional input space, you can use the PCA to reduce it to anywhere between 1 to N dimensions. So yes, you can reduce from 8 to 3; whether that makes any sense to do is up to your decision (based upon the concrete data in question).
