How to prove higher eigenvalues correspond to significant principal components? Is there any explanation or at least intuition for that, if not a proof?
I would like to convince myself with some intuition that eigenvalues of covariance matrix correspond to principal components and higher eigenvalues correspond to significant principal components etc.
Your data is a set of points in an n-space. Let's say that you have two variables, i.e. your data matrix is $N\times 2$. If there is no correlation the scatterplot is bounded by a circle, otherwise by an ellipse, the eigenvectors of the covariance matrix are parallel to the ellipse axes and their lengths are proportional to the square root of their respective eigenvalues. This is why the eigenvalues correspond to (directions and amounts of) variability. Look a this figure (R code below):
dat <- read.table("http://www.ces.clemson.edu/~calvinw/MthSc807/MthSc807/data/T3_7_SONS.DAT")
x <- dat$V1 - mean(dat$V1)
y <- dat$V2 - mean(dat$V2)
plot(x,y, xlim=c(-30,30), ylim=c(-30,30))
# ellipse
library(ellipse)
S <- cov(dat[,1:2])
lines(ellipse(S))
# eigenvectors and their length (square root of eigenvalues)
eig <- eigen(S)
lambda <- eig$values
a1 <- eig$vectors[,1]
a2 <- eig$vectors[,2]
arrows(0, 0, sqrt(lambda[1])*a1[1], sqrt(lambda[1])*a1[2], length=0.1)
arrows(0, 0, sqrt(lambda[2])*a2[1], sqrt(lambda[2])*a2[2], length=0.1)
If your data matrix is $N\times 3$ you get an ellipsoid, but if it is flat like a cuttlebon, then the two longer eigenvectors correspond to "principal components" of variability.
Parallel analysis may suit your purposes. It does not involve a null hypothesis significance test, and its results should not be confused with statistical significance in the usual sense of the term. However, it gives a fairly intuitive indication of how large eigenvalues should be at a minimum to make sense as indicators of valid dimensional constructs.
Parallel analysis simulates random samples of data from uncorrelated distributions that resemble your sample. It calculates the eigenvalues of these simulated data, and compares these to your sample's eigenvalues, which should be higher if your variables relate at all...at least on as many components as make sense to extract from your data. When your data's eigenvalues drop below the size of the eigenvalues of the simulated data, you can be fairly confident that those components describe too little variance to be meaningful and replicable.
The intuitiveness of this comparison derives from the fact that the simulated data come from distributions that have no common components (because they are uncorrelated). Thus any component extracted from those data only describes covariance due to sampling error. Again, parallel analysis is not a significance test per se; it won't tell you the probability that your eigenvalues are larger than they should be due to sampling error. It does give you a basis for comparison though: it indicates how big eigenvalues can be expected to grow due entirely to sampling error.
BTW, it's worth noting that parallel analysis has been referred to as "a method for determining significant principal components". I think this appeals to a less conventional understanding of "significant" – as I've said, one that is not based on the probability of the data given the null. See this:
Franklin, S. B., Gibson, D. J., Robertson, P. A., Pohlmann, J. T., & Fralish, J. S. (1995). Parallel analysis: A method for determining significant principal components. Journal of Vegetation Science, 6(1), 99–106. Retrieved from http://opensiuc.lib.siu.edu/cgi/viewcontent.cgi?article=1004&context=pb_pubs.
