3

What should I know about the instability of the Pearson product-moment correlation coefficient? When might I experience problems using this calculation?

I will quote the following Wikipedia article for some background information: http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient#Mathematical_properties

The Pearson correlation can be expressed in terms of uncentered moments. Since $μX = E(X)$, $$σX2 = E[(X − E(X))2] = E(X2) − E2(X)$$ and likewise for Y, and since

$$E[(X-E(X))(Y-E(Y))]=E(XY)-E(X)E(Y)$$ the correlation can also be written as

$$\rho_{X,Y}=\frac{E(XY)-E(X)E(Y)}{\sqrt{E(X^2)-(E(X))^2}~\sqrt{E(Y^2)- (E(Y))^2}}$$ Alternative formulae for the sample Pearson correlation coefficient are also available:

$$r_{xy}=\frac{\sum x_iy_i-n \bar{x} \bar{y}}{(n-1) s_x s_y}=\frac{n\sum x_iy_i-\sum x_i\sum y_i} {\sqrt{n\sum x_i^2-(\sum x_i)^2}~\sqrt{n\sum y_i^2-(\sum y_i)^2}}$$ The second formula above needs to be corrected for a sample:

$$r_{xy}=\frac{\sum x_iy_i-n \bar{x} \bar{y}}{(n-1) s_x s_y}=\frac{n\sum x_iy_i-\sum x_i\sum y_i} {\sqrt{(n-1)\sum x_i^2-(\sum x_i)^2}~\sqrt{(n-1)\sum y_i^2-(\sum y_i)^2}}$$ The above formula suggests a convenient single-pass algorithm for calculating sample correlations, but, depending on the numbers involved, it can sometimes be numerically unstable.

raco
  • 339
  • 1
  • 3
  • 11

1 Answers1

3

You can experience problems whenever a term like $\sum y_i^2$ or $\sum x_iy_i$ is very large, and yet close to the second term, potentially leading to a large loss in digits of accuracy when almost all of the significant digits cancel. In the case of the variance, it happens when the standard deviation is small compared to the mean.

It's possible to construct one-pass forms for all the terms under the $\sqrt{}$ signs that don't suffer this sort of problem.

There's an example calculation for a variance given here. Similar calculations for covariance can be done.

Community
  • 1
Glen_b
  • 257,508
  • 32
  • 553
  • 939
  • So I will only encounter problems if the standard deviation of X or Y is small relative to it's respective mean? Is there a rule of thumb for this level? If I do not expect my data to follow this pattern, then is the "naive" algorithm is sufficient? – raco Apr 17 '14 at 19:19
  • It's merely the most obvious and probably biggest issue. The space of ways other things might go wrong if things are sufficiently bad is quite big (e.g. if you have very large amounts of data or very large numbers, or very small numbers with fairly large numbers) -- pretty much any of those things - and many more - can be much more likely to cause problems with that formula. Rules of thumb aren't much good because there are so many other things it depends on (even the *order* of the data can matter - a lot - as to how much of a problem you have). ... ctd – Glen_b Apr 17 '14 at 23:28
  • ctd ... The only time I wouldn't avoid it are cases where you wouldn't need to ask at all; if it's not 100% obvious it's fine, don't use it. – Glen_b Apr 17 '14 at 23:30