5

(Remark: this is a "scholastic" question - I'm reviewing my implementation of factor analysis procedures; I'm not looking for good approximations for an actual survey/actual data or the like.)

There are different methods of estimating individual variances for the items in a covariance matrix $C$ with $m$ rows and columns; I know that method of using $D$, the reciprocal of the diagonal of the inverse of $C$. Well, it shall result in a Heywood-case/in negative definiteness of the remaining matrix if I simply remove that variance from the diagonal of the covariance matrix ($C - D$ is surely negative definite); but I can iteratively determine the greatest possible part in $D \cdot 1/r$ to be removed which still keeps the covariance $C - D/r$ positive semidefinite. This gives then a certain sum of that itemspecific variances ($s_1=sum(D/r)$).
Another method is to get the least principal axis $A_m$ , take the eigenvalue $\lambda _m $ , then norm the other axes to the same length $B_k = A_k \cdot \lambda_m / \lambda_k $ and in the diagonal of $ B \cdot B^T$ we get (equal) itemspecific variances. Note that again $ C - B \cdot B^T $ has reduced rank, and thus "all individual variance" is removed - however, the sum of all that itemspecific variances $s_2$ is usually much smaller than than $s_1$.
After that two different solutions, already leading to different amounts of overall itemspecific variance removed, I experimented with further different methods and one gives $s_3$ which is even greater than $s_1$.

And having now a handful of further methods with different values $s_j$, the question naturally occurs:

Q: is there a specific method, which allows to extract the maximally possible sum of individual variances of a covariance matrix, and if there is a special method, how is it defined?

To see, that the differences between the methods are not simply neglectable I add an example with some test-covariance matrix.
Overview, comparision of 4 methods:

enter image description here

Detail 1:
enter image description here

Detail 2: I'm surprised that the shape of the approaching of the maximum has such a spike - I'd expect some smooth "top of a normal-curve" here:
enter image description here

Gottfried Helms
  • 1,494
  • 15
  • 23
  • `Item-specific variance` Are you talking about what is known in FA as _unique variance_ (uniqueness) as opposed to _common variance_ (communality)? And you are playing with methods to estimate initial communalities before FA iterations start, aren't you? – ttnphns Nov 09 '12 at 11:14
  • @ttnphns: that is both correct, yes. – Gottfried Helms Nov 09 '12 at 11:22
  • I wonder why you want to chase and remove as much as possible of item-specific variance. That looks strange to me. Though, honestly, I don't follow your approach. – ttnphns Nov 09 '12 at 12:23

1 Answers1

8

Not sure my response is relevant, perhaps what I say is not news for you. It is about starting values for communalities in factor analysis.

Actually, you cannot estimate the true communality (and likewise uniqueness) of a variable before you've done FA. This is because communalities are tied up with the number of factors m being extracted. In Principal Axes factor analysis method of extraction communalities are being iteratively trained (like dogs are trained) to restore pairwise coefficients - correlations or covariances - maximally by m factors.

To estimate starting values for communalities several methods can be used, as you probably know:

  • The squared multiple correlation coefficient$^1$ between the variable and the rest variables is considered the best guess for the starting value of communality of the variable. This value is the lower bound for the "true", resultant, communality.
  • Another possible guess for the value is the maximal or the mean absolute correlation/covariance of the variable with the rest ones.
  • Still, another guess value used sometimes is the test-retest reliability (correlation/covariance) coefficient. This would be the upper bound for the "true" communality.
  • And in specific cases, user-defined initial values are used (e.g. communality values borrowed from literature).

$^1$ A closer look. If $\bf R$ is the analyzed correlation or covariance matrix, and you make diagonal matrix $\bf D$ with the diagonal elements being the inverses of diagonal elements of $\bf R^{-1}$, then matrix $\bf DR^{-1}D-2D+R$ is called "image covariance matrix" of $\bf R$ (sic! "covariance" irrespective whether $\bf R$ is covariances or correlations). Its diagonal entries are "images" in $\bf R$ (actually, these images are the diagonal of $\bf R-D)$.

If $\bf R$ is correlation matrix, images are the squared multiple correlation coefficients (of dependency of a variable on all the other variables). If $\bf R$ is covariance matrix, images are the squared multiple correlation coefficients multiplied by the respective variable variance. These values - the images - are used as starting communalities in both cases.

A side note for the curious: matrix $\bf DR^{-1}D$ is known as "anti-image covariance matrix" of $\bf R$. If you convert it to "anti-image correlation matrix" (in a usual way like you convert covariance in correlation, $r_{ij}=cov_{ij}/(\sigma_i \sigma_j)$), then the off-diagonal elements as a result are the negatives of partial correlation coefficients (between two variables controlled for all the other variables). Partial correlation coefficients are optionally used within factor analysis to compute Kaiser-Meyer-Olkin measure of sampling adequacy (KMO).

See also.

Community
  • 1
ttnphns
  • 51,648
  • 40
  • 253
  • 462
  • I'm sorry I left your answer (and also my question) aside in Nov and had to do other things and had partially bad health, so I forgot this completely. Now just to add a reaction to your effort to help me - see the next comment, this one might be deleted later. – Gottfried Helms Feb 24 '13 at 11:54
  • Well, I've looked at that problem of guessing initial variances from systematic view. As you state correctly, that the squared multiple covariance coefficient is the lower bound, it is also systematically too high and would lead to a Heywood-case. What I've done was to find a constant scaling $s^2<1$ for all of these estimates which lets the covariance/correlation matrix positive semidefinite and reduces just one rank, so the upper bound for such a scaling factor. But this gives only one specific estimate for the overall individual variance. I found, that scaling the estimates differently ... – Gottfried Helms Feb 24 '13 at 12:00
  • ... the overall individual variance can be improved. My question concerns the upper bound for the overall individual variance which is possible by any guess for the single individual variances. It's just to complete my mathematical models... – Gottfried Helms Feb 24 '13 at 12:02
  • @ttnphns: Thanks for the update. So the "image matrix" is only used to get its diagonal elements, which we plug into the diagonal of $\mathbf{R}$ to start the iterations, is that correct? Is there any meaning (or use) to the off-diagonal elements of the "image matrix"? – amoeba Jun 11 '14 at 15:33
  • @amoeba, I added a line. Image and anti-image matrices can be useful. – ttnphns Jun 11 '14 at 15:55
  • @ttnphns: Thanks! The reason I asked my question above, is that if one is only interested in the estimates of communalities, i.e. in the diagonal elements of the "image matrix", then your equation simplifies a lot. Diagonal of $\mathbf{D}\mathbf{R}^{-1}\mathbf{D}$ is equal to $\mathbf{D}$, so, if I am not mistaken, the formula reduces to $\mathbf{R}-\mathbf{D}$, which is equal to $\mathbf{R}$ with diagonal values replaced by communalities estimates (so $\mathbf{R}-\mathbf{D}$ can be directly used in the factor analysis iterations). – amoeba Jun 11 '14 at 16:02
  • @amoeba, yes, thank you! Images are indeed equal to the diagonal elements of $\bf R-D$, so here is the simplification. However, because partial correlations are used in computation of KMO, $\bf{D}\bf{R}^{-1}\bf{D}$ may be necessary. – ttnphns Jun 11 '14 at 16:25