4

I have the following problem that I am not sure of whether I treat it correctly:

I run repeated simulations, where the output of a single simulation replicate is a single correlation value. The simulation is replicated (let's say) 1,000 times and finally the mean correlation is calculated for a certain parameter setting of the simulation.

The goal is now to compare for different parameter settings not only the mean correlation, but also to which degree the correlation varies across simulation replicates in comparison with other parameter settings. My first approach was to compare the standard deviation of the correlation across the different parameter settings, but I am uncertain about how this is influenced by the actual mean value of the correlation. Would it be appropriate to use the coefficient of variation, i.e., SD(x)/Mean(x), x being the vector of the 1,000 correlation values of one parameter setting, and to compare this value to those of other parameter settings?

Thanks in advance for any help!

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
PScho
  • 41
  • 3

1 Answers1

1

The value of the correlation coefficient is of importance to qualify its uncertainty. It can be shown with the Cramer-Rao lower bound. In the particular case of a correlation coefficient, you can find results here: CRLB for Pearson linear correlation, slides 12 and 13. It means that the variance of an unbiased estimator cannot be lower than that. You can notice that the higher the correlation, the more precise the estimation can be. The coefficient you propose SD(x)/Mean(x) has the same idea in it (well done), but is totally ad hoc, and, to the best of my knowledge, not theoretically supported.

Also, be careful, what is the mean correlation? The arithmetic average of the estimated correlation values? There are arguments (cf. information geometry) to show that it is not a good definition. For example, in the multivariate case, an arithmetic average of correlation matrices will not even yield a correlation matrix: there are good chance that the resulting matrix is not positive semi-definite.

If you want to work further these questions, you can google "information geometry of the multivariate normal" to have a glimpse of what you can/should do. Good luck!

mic
  • 3,848
  • 3
  • 23
  • 38
  • 1
    Thanks a lot. I get the basic idea of what you are saying concerning the boundry of the variance of the estimator. I am in fact using the arithmetic mean of the (1,000) correlation estimates and was not aware that this might even be problematic (common practice in my field of research). So are you saying using the coefficient of variation of the correlation estimates might be a useful approach to compare different populations of estimates (i.e. different and independent sets of n=1,000 estimates) or at least get an idea about their difference? – PScho Aug 31 '16 at 14:35
  • I do not know if your coefficient SD(x)/Mean(x) can be helpful in your case. I'm just saying that it is not well supported in theory. What you can do is using for example the Kullback-Leibler divergence (close formula can be found in wikipedia) between two bivariate centered Gaussians (parametrized by your estimated average correlation), and you can use the dissimilarity it gives. Kullback-Leibler (Hellinger, and others called the $f$-divergences) are in fact locally taking into account the CRLB: when correl is low, the divergence grows slowly, high correl yields fast divergence. – mic Aug 31 '16 at 14:44