Correct equation for weighted unbiased sample covariance

Question

I'm looking for the correct equation to compute the weighted unbiased sample covariance. Internet sources are quite rare on this theme and they all use different equations.

The most likely equation I've found is this one:

$q_{jk}=\frac{\sum_{i=1}^{N}w_i}{\left(\sum_{i=1}^{N}w_i\right)^2-\sum_{i=1}^{N}w_i^2} \sum_{i=1}^N w_i \left( x_{ij}-\bar{x}_j \right) \left( x_{ik}-\bar{x}_k \right) .$

From: https://en.wikipedia.org/wiki/Sample_mean_and_sample_covariance#Weighted_samples

Of course, you have to compute the weighted (unbiased) sample mean beforehand.

However, I have found several other formulas like:

$q_{jk}= \frac{1}{\sum_{i=1}^N w_i)-1}\sum_{i=1}^N w_i \left( x_{ij}-\bar{x}_j \right) \left( x_{ik}-\bar{x}_k \right) .$

Or I've even seen some source codes and academic papers just using the standard covariance formula but with the weighted sample mean instead of the sample mean...

Can someone help me and shed some light?

/EDIT: my weights are simply the number of observations for a sample in the dataset, thus weights.sum() = n

What kinds of weights are you using? Weights can mean different things, according to the application, and the correct answer depends on their meaning. For instance, they can be shorthand for frequencies (an observation has a weight of $f$ in the dataset to reflect its occurrence $f$ times in the data) or they can be shorthand for probabilities of being in a sample (for weighted random samples, such as stratified or hierarchical samples). — whuber, Jun 08 '13 at 18:57
@whuber: My weights are simply the number of observations for one sample in the dataset, and thus: Sum(weights) = n — gaborous, Jun 08 '13 at 23:01
Then your question is answered in many places here (although that might not immediately be apparent), including http://stats.stackexchange.com/questions/58986/how-to-compute-weighted-means-and-sds (which explains the principle) as well as http://stats.stackexchange.com/questions/6534/how-do-i-calculate-a-weighted-standard-deviation-in-excel/6536#6536 (which I still maintain is incorrect, but you can decide for yourself). — whuber, Jun 09 '13 at 13:41
@whuber: thank you for your help, but the first link is about standard deviation (squared root of variance) and not about covariance, and the second one is plainly wrong (as you pointed). — gaborous, Jun 09 '13 at 16:43
And if you don't take the square root in the first one, you get the variance... — whuber, Jun 09 '13 at 17:01
@whuber: **covariance** is what I need, not just variance.... — gaborous, Jun 09 '13 at 17:15
There is no difference between variance and covariance: all covariances can be obtained as (linear combinations) of variances and variances, of course, are just particular examples of covariances. (Mathematicians call this relationship [polarization](http://en.wikipedia.org/wiki/Polarization_identity).) — whuber, Jun 10 '13 at 13:01
@whuber: technically I agree, but since covariance is a generalization of variance, I needed to make sure that I didn't miss something in the equation I derived. — gaborous, Jun 10 '13 at 18:11
@whuber: And thank's for the pointer to relationship polarization, that will be very helpful. — gaborous, Jun 10 '13 at 18:40

gaborous · Accepted Answer · 2021-01-31T05:07:32.947

22

Found the solution in a 1972's book (George R. Price, Ann. Hum. Genet., Lond, pp485-490, Extension of covariance selection mathematics, 1972).

Biased weighted sample covariance:

$\Sigma=\frac{1}{\sum_{i=1}^{N}w_i}\sum_{i=1}^N w_i \left(x_i - \mu^*\right)^T\left(x_i - \mu^*\right)$

And the unbiased weighted sample covariance given by applying the Bessel correction:

$\Sigma=\frac{1}{\sum_{i=1}^{N}w_i - 1}\sum_{i=1}^N w_i \left(x_i - \mu^*\right)^T\left(x_i - \mu^*\right)$

Where $\mu^*$ is the (unbiased) weighted sample mean:

$\mathbf{\mu^*}=\frac{\sum_{i=1}^N w_i \mathbf{x}_i}{\sum_{i=1}^N w_i}$

Important Note: this works only if the weights are "repeat"-type weights, meaning that each weight represent the number of occurrences of one observation, and that $\sum_{i=1}^N w_i=N^*$ where $N^*$ represent the real sample size (real total number of samples, accounting for the weights).

I have updated the article on Wikipedia, where you will also find the equation for unbiased weighted sample variance:

https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_covariance

Practical note: I advise you to first multiply column-by-column $w_i$ and $\left(x_i - \mu^*\right)$ and then do a matrix multiplication with $\left(x_i - \mu^*\right)$ to wrap things up and automatically perform the summation. Eg in Python Pandas/Numpy code:

import pandas as pd
import numpy as np
# X is the dataset, as a Pandas' DataFrame
mean = mean = np.ma.average(X, axis=0, weights=weights) # Computing the weighted sample mean (fast, efficient and precise)
mean = pd.Series(mean, index=list(X.keys())) # Convert to a Pandas' Series (it's just aesthetic and more ergonomic, no differenc in computed values)
xm = X-mean # xm = X diff to mean
xm = xm.fillna(0) # fill NaN with 0 (because anyway a variance of 0 is just void, but at least it keeps the other covariance's values computed correctly))
sigma2 = 1./(w.sum()-1) * xm.mul(w, axis=0).T.dot(xm); # Compute the unbiased weighted sample covariance

Did a few sanity checks using a non-weighted dataset and an equivalent weighted dataset, and it works correctly.

For more details about the theory of unbiased variance/covariance, see this post.

edited Jan 31 '21 at 05:07

answered Jun 09 '13 at 17:13

gaborous

635
1
8
22

@whuber: no it's correct, but it depends on what your "weights" are assigned to. In my case, it's the number of observations (aka "repeats"), as such this equation correctly works. In your case with normalized weights or also if the weights are the variances of each measurement of an observation, so-called "reliability", then it doesn't work and the other equation on the wiki should be used (which btw doesn't work with "repeats"-type weights!). – gaborous Jun 10 '13 at 18:15
@whuber: anyway if you have a more generalizable approach, I would be glad to hear about it. I have crawled the whole internet and this website and didn't find an equation that would work for repeats-type weights except this one I have posted above! – gaborous Jun 10 '13 at 18:16
3

There cannot possibly be a universal formula: if the weights are not integral frequencies (for instance, if they have been standardized to sum to unity), then you have lost all information about the total sample size $n$, whence it is impossible to estimate the correction factor $n/(n-1)$. Nevertheless, provided the weights do not sum to unity, your formula will produce an answer. That is why it's crucial to make clear that your formula is *not* a generally applicable one and that the $w_i$ *must* be actual frequencies and not anything else. – whuber Jun 10 '13 at 18:20
3

@whuber: Thank's for the explanation, that's what I feared (no unified equation and loss of correction factor). I have added a note to my answer to better describe this. I am also going to add your line about the correction factor on the Wikipedia article. – gaborous Jun 10 '13 at 18:39
@whuber Came across these old threads. I am wondering how your statement that "there cannot possibly be a universal formula" fits to e.g. an accepted answer in this thread: http://stats.stackexchange.com/questions/47325. Do you think that answer is wrong (in claiming that there is an unbiased expression for any weights)? – amoeba Feb 06 '17 at 19:50
1

@Amoeba I believe the controversy played out in the comments makes my point well: there are different kinds of weights that call for different formulas. – whuber Feb 06 '17 at 20:00
1

@whuber: What is your take on the formula that is used by https://numpy.org/doc/stable/reference/generated/numpy.cov.html? Isn't that general? – Make42 Apr 30 '21 at 15:06
3

@Make42 It is not general. There are different kinds of weights that can enter into formulas differently. Some weights adjust for different levels of imprecision (variance weights); others for different inclusion probabilities in a sample (probability weights); as a shorthand for specifying multiple observations (frequency weights); and yet others for inducing arbitrary weighting to give greater importance to specified observations in a dataset. – whuber Apr 30 '21 at 15:54

Correct equation for weighted unbiased sample covariance

1 Answers1

Linked