Why is the maximum risk of an estimator independent of a prior distribution over the parameter?

Question

One way of choosing an estimator $\delta(x)$ for data $X$ distributed as $P_{\theta}(X)$, where $\theta \in \Theta$ is:

$$minimize \sup_{\theta \in \Theta} Risk(\delta(x), \theta)$$

In this case why is this risk independent of a prior over $\theta$?

Answer 1

It would be so comforting to find an estimator that always does better than any other, as it would end up the debate on which estimator to pick! Sadly, it is almost never the case that there exists a uniformly best estimator.

The approach in the question is called minimaxity, which compares estimators in terms of their worst performance, $$\sup_{\theta\in\Theta} R(\delta,\theta)=\sup_{\theta\in\Theta} \Bbb E_\theta[L(\delta(X),\theta)]$$ Since this quantity is a real number all estimators become comparable under this criterion and hence there may exist a solution to the minimisation program $$\inf_\delta \sup_{\theta\in\Theta} R(\delta,\theta)$$ At no stage in this formalisation does a prior occur because (a) estimators can be constructed from any point of view, not necessarily Bayesian, and (b) the distribution involved in the computations is purely frequentist, i.e. a distribution in $X$. Further, considering the supremum in $\theta$ cancels the relevance of a prior distribution over $\theta$.

Note however that in regular problems, maximin and minimax coincide (see e.g. in my book, Section 2.4.3), that is $$\inf_\delta \sup_{\theta\in\Theta} R(\delta,\theta)=\sup_\pi\inf_\delta \int R(\delta,\theta)\,\pi(\theta)\text{d}\theta.$$This means that, when the problem is regular enough, the minimax estimator is also Bayes against a prior called least favourable prior.

Here is an example taken from Berger (1985)

Consider $x\sim{\mathcal B}(n,\theta)$ when $\theta$ is to be estimated under the quadratic loss, $$ \mathrm L (\theta,\delta) = (\delta-\theta)^2. $$ Bayes estimators are then given by \post\ expectations (see Section 2.4) and, when $$\theta \sim{\mathcal B}e \left(\frac{\sqrt{ n}}{2}, \frac{\sqrt{ n}}{2} \right)$$ the posterior mean is $$ \delta^* (x) = \frac{x+ \sqrt{n}/2}{n+ \sqrt{ n}}. $$ Moreover, this estimator has constant risk, $$R(\theta,\delta^*) = 1/4(1+\sqrt{n})^2$$ Therefore, integrating out $\theta$, $$r(\pi) = R(\theta,\delta^*)$$ and this shows $\delta^*$ is minimax according to Lemma 2.4.13. Notice the difference with the maximum likelihood estimator, $$\delta_0(x) = x/n$$ for small values of $n$, and the unrealistic concentration of the prior around $0.5$ for larger values of $n$.