Which is a better estimator, averaged functions vs. A function of an average?

Question

Problem:

Assume that we want to estimate $f(\theta)$ with a pre-specified strictly increasing function $f$ and a parameter $\theta$.

Let $\hat{\theta}_1$ and $\hat{\theta}_2$ be unbiased estimators for $\theta$. My question is to compare which estimator is better for $f(\theta)$:

1. $f\left(\frac{\hat{\theta}_1 + \hat{\theta}_2}{2}\right)$, or
2. $\frac{f(\hat{\theta}_1) + f(\hat{\theta}_2)}{2}$.

My approach:

My approach is taking an expectation on both based on Taylor's expansion of $f$, provided that they are well-defined.

1. $$\mathbb{E}f\left(\frac{\hat{\theta}_1 + \hat{\theta}_2}{2}\right) = f(\theta) + \frac{f''(\theta)}{2}\text{Var}\left(\frac{\hat{\theta}_1 + \hat{\theta}_2}{2}\right) + \ldots $$

2. $$\mathbb{E} \left[\frac{f(\hat{\theta}_1) + f(\hat{\theta}_2)}{2} \right] = f(\theta) + \frac{f''(\theta)}{2}\left(\frac{\text{Var}(\hat{\theta}_1) + \text{Var}(\hat{\theta}_2)}{2}\right) + \ldots $$

Here, the first-order disappears in each expectation because $\hat{\theta}_1$ and $\hat{\theta}_2$ are unbiased.

Let's further assume $f''(\theta)>0$. Then, since $$\label{varineq} (0 \le) \text{Var}\left(\frac{\hat{\theta}_1 + \hat{\theta}_2}{2}\right) \le \frac{\text{Var}(\hat{\theta}_1) + \text{Var}(\hat{\theta}_2)}{2}, $$ the first estimator (i.e. a function of averages) seems less distant from $f(\theta)$, as long as the remaining terms (order $\ge 3$) are ignored. The last inequality comes from a simple observation: $$ \frac{\text{Var}(\hat{\theta}_1) + \text{Var}(\hat{\theta}_2)}{2} - \text{Var}\left(\frac{\hat{\theta}_1 + \hat{\theta}_2}{2}\right) = \dfrac{1}{4}\left( \text{Var}(\hat{\theta}_1) + \text{Var}(\hat{\theta}_2) - 2\text{Cov}\left(\hat{\theta}_1, \hat{\theta}_2\right) \right) = \dfrac{\text{Var}(\hat{\theta}_1 - \hat{\theta}_2)}{4} \ge 0. $$

Question:

I wonder if this is logical and there is another way to justify a better estimator.

Edit:

Let's restrict the class of $f$ by the strictly increasing ones.

Answer 1

Suppose $\theta_1$ and $\theta_2$ are unbiased estimators of $\mu$ with a bivariate normal distribution.

Let $\sigma_1 = k \sigma_2$, with $k<1$, so $\theta_1$ is the more efficient estimator.

Let the correlation be $r$, with $-1<r<1$.

We seek the least-variance linear estimates for $\mu$ and $\mu^2$. The results are:

\begin{align} \min_{a,b}Var[\mu-(a\theta_1+b\theta_2)] \text{ has }&\mu \sim \frac{(1-kr)\theta_1+(k^2-kr)\theta_2}{1-2kr+k^2}\\ \min_{c,d,e}Var[\mu^2-(c\theta_1^2+d\theta_1\theta_2+e\theta_2^2)] \text { has }&\mu^2 \sim \left(\frac{(1-kr)\theta_1+(k^2-kr)\theta_2}{1-2kr+k^2}\right)^{\!2} \end{align}

So for functions $f$ that are quadratic or well approximated by quadratics, if $\frac{\theta_1+\theta_2}{2}$ is the best linear estimate for $\mu$, then $f\left(\frac{\theta_1+\theta_2}{2}\right)$ is the best linear estimate for $f(\mu)$. In general an appropriate weighted average inside $f$ works even better.

Example: $\theta_1$ is the mean of a sample from a normal parent with mean $\mu$, and $\theta_2$ is the median from the same sample. Then $k=r=\sqrt{2/\pi}$, and the above formula shows that the best estimate for $\mu$ is $\theta_1$, and the best estimate for $\mu^2$ is $\theta_1^2$, i.e. ignoring the median.

Example: $\theta_1$ is the mean of a sample from a normal parent with mean $\mu$, and $\theta_2$ is the median of a different sample of the same size. Then $k=\sqrt{2/\pi}$, $r=0$, so the best estimate for $\mu$ is $(\pi\theta_1+2\theta_2)/(\pi+2)$, or $.611\theta_1 + .389\theta_2$, and the best estimate for $\mu^2$ is the square of that.

Example: $\theta_1$ is the mean of a sample from a normal parent with mean $\mu$, and $\theta_2$ is the mean of a different sample of the same size. Then $k=1$, $r=0$, so the best estimate for $\mu$ is $(\theta_1+\theta_2)/2$, and the best estimate for $\mu^2$ is $\left(\frac{\theta_1+\theta_2}2\right)^{\!2}$.