Does the biased estimator always have less variance than unbiased one?

Question

Suppose I am estimating one of the parameter. Now if we plot the biased estimator of that and unbiased estimator of that can we say for sure that biased one has less variance than unbiased one always.

My thought: my analysis somehow tells me that it is true because if we check bias variance trade off then obviously if we increase the bias from 0 to something, that amount will be deduced from the variance part making the estimator peaky.

Answer 1

NO

Remember that just about anything can be an estimator, even silly estimators. Let’s consider two estimators for $k$ of $\chi^2_k$. Take an $iid$ sample $X_1,\dots,X_n$.

$$ \hat k_1=\bar X \\ \hat k_2=\sum_{i=1}^nX_i=n\bar X $$

$\hat k_1$ is unbiased, while $\hat k_2$ is biased. However, what are the variances?

$$ \mathbb{V}(\hat k_1)=2k/n\\ <\\ \mathbb{V}(\hat k_2)= 2nk \\ \forall n>1 $$

There’s your counterexample. $\square$

However, I see you making at least two mistakes in your setup.

1. Multiple estimators can be unbiased. The sample mean, sample median, and first observation (NOT first order statistic) are unbiased estimators for the mean of a normal distribution, for example. (Indeed, the $j^{th}$ observation (NOT order statistic) drawn from a distribution is an unbiased estimator for the mean whenever the distribution has a mean.)

2. The MSE does not have to be the same for biased and unbiased estimators. In fact, we tend to pick biased estimators over unbiased estimators because there is such a reduction in variance that the MSE decreases.

EDIT

An even easier example where we’re estimating $\mu$ of $N(\mu, \sigma^2)$:

$$ \hat\mu_1=\bar X\\ \hat\mu_2=\bar X+ 1 $$

Both have the same variance, yet only $\hat \mu_1$ is unbiased.

EDIT 2

If two estimators of a parameter $\theta$, one biased $(\hat\theta_1)$ by some amount $b$ and one unbiased $(\hat\theta_2)$, have the same $MSE$, then it must be that the biased estimator has lower variance. Let $MSE(\hat\theta_1) = MSE(\hat\theta_2) = M$.

$$ (\text{bias}(\hat\theta_1))^2 + \mathbb{Var}(\hat\theta_1) = M = (\text{bias}(\hat\theta_2))^2 + \mathbb{Var}(\hat\theta_2) $$$$ b^2 + \mathbb{Var}(\hat\theta_1) = 0 + \mathbb{Var}(\hat\theta_2) $$$$ \mathbb{Var}(\hat\theta_1) = \mathbb{Var}(\hat\theta_2) - b^2$$$$ \implies\mathbb{Var}(\hat\theta_1) > \mathbb{Var}(\hat\theta_2) $$

(I confess that I don't know what happens if we work over complex numbers (so that $b^2<0$ is possible), though I would imagine that the decompision of $MSE$ in that case is something like $MSE(\hat\theta) = (\text{bias}(\hat\theta)) \overline{(\text{bias}(\hat\theta))} + \mathbb{Var}(\hat\theta)$.)

Answer 2

There are many, many, many different possible estimators in estimation problems. In general there are multiple unbiased estimators and multiple biased estimators, and their variances need to be considered on a case-by-case basis. A biased estimator may have a lower, or higher, or the same variance as an unbiased estimator. Now, if we consider estimators on the basis of their mean-squared-error (MSE), then we will generally consider the lower bias and higher variance as a trade-off and we will ignore any estimators that have both higher bias and higher variance than another estimator. In this context we are left with estimators where the unbiased ones have higher variance and the biased ones have lower variance. Nevertheless, this is a contextual result that comes from setting other estimators out of consideration.

---

As a counter-example to show that it is possible to have an unbiased estimator with higher variance than a biased estimator, consider a set of data values $X_1,...,X_n \sim \text{N}(\mu, 1)$ and consider the estimators:

$$\hat{\mu}_1 \equiv X_1 + b \quad \quad \quad \quad \quad \hat{\mu}_2 \equiv \bar{X}_n,$$

where $b \neq 0$. It is simple to establish that $\hat{\mu}_1$ is biased and $\hat{\mu}_2$ is unbiased, and their respective variances are:

$$\mathbb{V}(\hat{\mu}_1) = 1 \quad \quad \quad \quad \quad \mathbb{V}(\hat{\mu}_2) = \frac{1}{n}.$$

If $n>1$ (i.e., if you have more than one data point) then the first estimator has a higher variance. Since this is the biased estimator, it is both biased and has higher variance than the unbiased estimator.

Answer 3

I don't think there's an answer as to which will have more variance. Take the case of the sample variance. The unbiased estimate

is $\sum_i (x_{i} - \bar{x})^2/(n-1)$. It has variance

$\left(\frac{1}{n-1}\right)^2$ Var$\left(\sum_i (x_{i} - \bar{x})\right)^2$

The biased estimate has variance:

$\left(\frac{1}{n}\right)^2$ Var$\left(\sum_i (x_{i} - \bar{x})\right)^2$

So, in this case, the biased estimate will have less variance. Each case probably depends on the estimate and a generalization as to which has or more less variance is probably not possible.

Answer 4

Yes

Provided, that the bias is decreasing the variance of the error.

In that case, since the bias reduces the variance of the error, which can be decomposed into contributions from bias and variance, the variance must be decreasing.

$$\text{var}(error) = bias(estimator)^2 + \text{var}(estimator)$$

So if you reduce $\text{var}(error)$ while increasing $bias(estimator)^2$ then necessarily $\text{var}(estimator)$ must decrease.

This means that the only sensible types of bias (sensible meaning that it reduces the error) are the ones that reduce the variance of the estimator. As explained below this is not true for every type of biased estimator.

---

No

However, if one uses some silly estimator with enormous variance and that has also bias, then sure a biased estimator can have more variance.

Example, related question

This is related to the question

Why exactly $E[(\hat{\theta}_n - E[\hat{\theta}_n])^2]$ and $E[\hat{\theta}_n - \theta]$ cannot be decreased simultaneously?

Seen here: Bias / variance tradeoff math

In an answer to that question we see the following graph for a shrinking estimator.

We can estimate the mean of some population $\mu$ by the sample mean $\bar{x}$ and then multiply it with some scaling factor $c\bar{x}$.

In the graph on the right you see what happens with variance and bias when the shrinking (or inflating) parameter is changed. The unbiased estimator is in the middle ($c=1$). You can add bias by multiplying with a factor below one ($c<1$to the left of the graph) or with a factor above one ($c>1$to the right of the graph).

You can see that multiplying with a factor above one is not decreasing the variance of the estimator (obviously since the variance of the estimator scales with $c^2$).

But, this is also not a type of bias that decreases the variance of the error, and it is not a type of bias that is typically considered in a bias-variance trade-off.

Answer 5

It is certainly not the case that all biased estimators have less variance than unbiased ones. That is quite trivial to see: pick any unbiased estimator, then create a new estimator that consists of that estimator plus some random non-zero mean random variable. It is also not the case the lowest variance biased estimator always has lower variance than the lowest variance unbiased estimator. For instance, given standard linear regression assumptions, OLS gives a zero bias estimator that has the lowest overall variance.

However, if the lowest variance unbiased estimator has a lower variance than the lowest variance biased estimator, then there probably isn't any reason to use the biased estimator, so you aren't going to hear about the biased estimator. That is, if estimator with the lowest variance overall is unbiased, then people are going to just use that one, and there's no reason to examine which biased estimators have the lowest variance. So ... I guess you can say that you have a biased view of biased estimators, because you only hear about the ones that have lower variance than the unbiased estimator.