5

Suppose I have a number of samples drawn from a normal distribution $x_i \sim \mathcal{N}(\mu,C)$ with $i = 1 \dots n$. I can make observations $z_i = x_i + e_i$ for those samples which are perturbed by noise $e_i$ of known characteristic $e_i \sim \mathcal{N}(0,\Sigma_i)$. Note that the distribution of the observation noise is different for each $z_i$.

I would like to estimate $\mu$ and $C$ given the $z_i$.

Since $e_i$ is unbiased $\mu$ should just be $\mathbb{E}[z_i]$. In case $\Sigma_i = D$ for all $i$, $C = \operatorname{Var}(z_i) - D$ assuming independence between the noise. Would it be safe to say that $C = \operatorname{Var}(z_i) - \mathbb{E}[\Sigma_i]$ in the general case?

Jakob
  • 281
  • 1
  • 8
  • (1) What does $\mathbb{E}[\Sigma_i]$ mean when it is stipulated that $\Sigma_i$ is a "known characteristic"? Are you suggesting there is some prior distribution involved? (2) Because there is no guarantee that $\text{Var}(z_i) - \mathbb{E}[\Sigma_i]$ is positive definite, you might want to worry a bit about the potential inadmissibility or unrealism of your estimator. – whuber Feb 19 '13 at 21:28
  • (1) $\mathbb{E}[\Sigma_i] = 1/n \sum_{i=1..n} \Sigma_i$ (2) The same would be true for when $\Sigma_i = D$ for all $i$ as well. How do I get around this problem? And yes, I worry about the admissibility of the estimator, hence my question :) – Jakob Feb 19 '13 at 21:51
  • If $\Sigma_i$ is known, and $\Sigma_i = L_i L_i'$, if you let $x_i^* = L_i^{-1} x$ (and similarly for $z$ and $e$), then depending on how you tend to look at things that may help a bit. – Glen_b Feb 19 '13 at 22:06
  • @whuber I've looked at [your answer](http://stats.stackexchange.com/a/12255/19731) to a similar problem, basically performing a weighting of the samples for the estimators. Which I guess is similar to what Glen_b is suggesting, right? I can't seem to derive an good estimator for $C$ based on the samples though. – Jakob Feb 20 '13 at 11:30
  • Jakob, are the $x_i$ numbers or *vectors*? Your use of capital $\Sigma$ for the variance suggests you are considering them vectors but your edit makes sense only if they are numbers. – whuber Feb 20 '13 at 16:43
  • 1
    @whuber Ah sorry. Currently I am happy with $x_i$ being univariate, but I think I might have to extend it to the multivariate case later. I did an update to the notation to make it more clear, but reverted it again since then its out of line with the comments. – Jakob Feb 20 '13 at 19:46

1 Answers1

2

I think I've figured it out now. As stated in the comments, I think there is indeed a problem with the estimators. I found another answer on using a linear estimator for a similar problem. If I didn't make any mistakes, the estimator for $\mu$ in my case should be $$\hat{\mu} = \frac{ \sum_{i=1..n} z_i/(C+\Sigma_i)}{ \sum_{i=1..n} 1/(C+\Sigma_i)}$$ Since I don't actually know $C$, I guess I could use the estimate $\hat{C}$ based on the samples. Based on the Wikipedia Article on weighted mean and then subtracting the covariance influence from the $\Sigma_i$, I can estimate the covariance as $$\hat{C} = \left(\frac{\sum w_i}{(\sum w_i)^2 - \sum w_i^2} \sum w_i(z_i - \hat{\mu}) \right) - \frac{n}{\sum w_i}$$ where $w_i = 1/\Sigma_i$.

Community
  • 1
Jakob
  • 281
  • 1
  • 8