Minimum Mean Square Estimator - Equivalent Expressions to Minimize

Question

Given $ M \in \mathbb{R}^{N \times N} $ which is a Positive Definite Matrix.
Let $ \hat{x} $ the MMSE of $ x $ given $ z $, namely $ \hat{x} = \mathbb{E} \left[ x \mid z \right] $.

Prove the equivenalce of the following expressions:

1. $ \arg \min_{\hat{x}} \mathbb{E} \left[ {\left( \hat{x} - x \right)}^{T} \left( \hat{x} - x \right) \mid z \right] $.

2. $ \arg \min_{\hat{x}} \mathbb{E} \left[ {\left( \hat{x} - x \right)}^{T} M \left( \hat{x} - x \right) \mid z \right] $.

3. $ \arg \min_{\hat{x}} \operatorname{Tr} \left( M \mathbb{E} \left[ {\left( \hat{x} - x \right)}^{T} \left( \hat{x} - x \right) \mid z \right] \right) $

Equivalence means all are actually minimized by $ \hat{x} = \mathbb{E} \left[ x \mid z \right] $.

The equivalence of 2 and 3 is easy using the Cyclic Property of the $ \operatorname{Tr} \left( \cdot \right) $ operator.
Yet showing the invertible transformation keeps the solution (Equivalence of 1 and 2) isn't trivial.

Answer 1

Since $ M \in \mathbb{S}^{N}_{++} $ (In other convention $ M \succ 0 $) by Cholesky Decomposition there is a Triangular Matrix $ R \in \mathbb{R}^{N \times N} $ such that $ M = {R}^{T} R $.

Using this fact one could prove $ 1 \iff 2 $ as following:

$$\begin{align*} \arg \min_{\hat{x}} \mathbb{E} \left[ {\left( \hat{x} - x \right)}^{T} M \left( \hat{x} - x \right) \mid z \right] & = \arg \min_{\hat{x}} \mathbb{E} \left[ {\left( \hat{x} - x \right)}^{T} {R}^{T} R \left( \hat{x} - x \right) \mid z \right] && \text{$M = {R}^{T} R $} \\ & = \arg \min_{\hat{x}} \mathbb{E} \left[ {\left( R \left( \hat{x} - x \right) \right)}^{T} \left( R \left( \hat{x} - x \right) \right) \mid z \right] && \text{} \\ & = \arg \min_{\hat{x}} \mathbb{E} \left[ {\left( R \hat{x} - y \right)}^{T} \left( R \hat{x} - y \right) \mid z \right] && \text{Defining $ y = R x $} \end{align*}$$

Clearly, the above, as the classic MMSE problem, with respect to $ y $ given $ z $ which is minimized when:

$$ R \hat{x} = \mathbb{E} \left[ y \mid z \right] \Rightarrow \hat{x} = {R}^{-1} \mathbb{E} \left[ y \mid z \right] $$

Since $ M \succ 0 $ then $ {R}^{-1} $ is defined and the above is valid.
Moreover by linearity of the Expectation Operator:

$$ \hat{x} = {R}^{-1} \mathbb{E} \left[ y \mid z \right] = {R}^{-1} \mathbb{E} \left[ R x \mid z \right] = {R}^{-1} R \mathbb{E} \left[ x \mid z \right] = \mathbb{E} \left[ x \mid z \right] $$

Hence the minimizer of 1 indeed minimizes 2.

Using the cyclic property of the Trace Operator and the linearity of the Expectation Operator (Hence one could change the order of the $ \operatorname{Tr} \left( \cdot \right) $ and $ \mathbb{E} \left[ \cdot \right] $) one could easily show $ 2 \iff 3 $ hence the problem is solved.

Answer 2

Use the Cholesky decomposition of $M$ as a change of coordinates.

$tr(ABC) = tr(CAB)$ - trace is invariant under cyclic permutations, and $tr(scalar)=scalar$. Thus, $tr(M E[ x x^T]) = tr(E[M x x^T]) = E[tr(M x x^T) ] = E[tr( x^T M x ) ] = E[x^T M x]$. Now, replace $x$ with $\hat{x}-x$.