Bayesian and frequency tail estimation

Question

The tail probability can be estimated by two methods:

1. In Bayesian method: $$P_B(X>a)=\int^{\infty}_{-\infty}\pi(\theta|x)[1-F(a|\theta)]d\theta$$
2. In Plug-in frequency method: $$P_F(X>a)=1-F(a|\hat{\theta})$$ where $\hat{\theta}$ is the MLE of $\theta$.

The numerical results show that it's always $$P_B \geq P_F$$ no matter what the distribution is.

Any ideas or any resources related to this topic to explain why is that?

Answer 1

First note that $P_B(X > a)$ is the value of the survival function of the predictive distribution, which is a mixture of distributions taken in the parametric family considered, the mixture weight being the posterior density.

I do not think that the result is always true, that is: for any distribution, for any sample, and for any $a$. However, the predictive distribution has tail which is thicker than any parametric distribution in the posterior which means that the inequality holds for large $a$. Under quite general conditions, the ML estimate $\hat{\theta}$ is in the support of the posterior, and the claimed relation should hold for large $a$. The reason is that a mixture of continuous distributions has a tail which is heavier than that of its components. I wrote some details in this answer to an apparently quite different question.

To get a counterexample we can choose a prior such that the ML estimate $\hat{\theta}$ does not always belong to it. For example consider an exponential distribution with mean $\theta$ assumed to lie in $(0, \, 1)$. If it happens that $\hat{\theta} > 1$ then the inequality will not hold.