Example of how Bayesian Statistics can estimate parameters that are very challenging to estimate through frequentist methods

Question

Bayesian statisticians maintain that "Bayesian Statistics can estimate parameters that are very challenging to estimate through frequentist methods". Does the following quote taken from this SAS documentation says that same thing?

It provides inferences that are conditional on the data and are exact, without reliance on asymptotic approximation. Small sample inference proceeds in the same manner as if one had a large sample. Bayesian analysis also can estimate any functions of parameters directly, without using the "plug-in" method (a way to estimate functionals by plugging the estimated parameters in the functionals).

I saw a similar statement in some textbook but do not recall where. Can anyone please explain this to me with an example?

Answer 1

I have objections with that quote:

1. "Frequentism" is an approach to inference that is based on the frequency properties of the chosen estimators. This is a vague notion in that it does not even state that the estimators must converge and if they do under how they must converge. For instance, unbiasedness is a frequentist notion but it cannot hold for any and every function [of the parameter $\theta$] of interest since the collection of transforms of $\theta$ that allow for an unbiased estimator is very restricted. Further, a frequentist estimator is not produced by the paradigm but must first be chosen before being evaluated. In that sense, a Bayesian estimator is a frequentist estimator if it satisfies some frequentist property.
2. The inference produced by a Bayesian approach is based on the posterior distribution, represented by its density $\pi(\theta|\mathfrak{D})$. I do not understand how the term "exact" can be attached to $\pi(\theta|\mathfrak{D})$.It is uniquely associated with a prior distribution $\pi(\theta)$ and it is exactly derived by Bayes theorem. But it does not return exact inference in that the point estimate is not the true value of the parameter $\theta$ and it produces exact probability statements only within the framework provided by the pair prior x likelihood. Changing one term in the pair does modify the posterior and the inference, while there is no generic argument for defending a single prior or likelihood.
3. Similarly, other probability statements like “the true parameter has a probability of 0.95 of falling in a 95% credible interval” found in the same page of this SAS documentation has a meaning relative to the framework of the posterior distribution but not in absolute value.
4. From a computational perspective, it is true that a Bayesian approach may often return exact or approximate answers in cases when a standard classical approach fails. This is for instance the case for latent [or missing] variable models$$f(x|\theta)=\int g(x,z|\theta)\,\text{d}z$$where $g(x,z|\theta)$ is a joint density for the pair $(X,Z)$ and where $Z$ is not observed, Producing estimates of $\theta$ and of its posterior by simulation of the pair $(\theta,\mathfrak{Z})$ may prove much easier than deriving a maximum likelihood [frequentist?] estimator. A practical example of this setting is Kingman's coalescent model in population genetics, where the evolution of populations from a common ancestor involves latent events on binary trees. This model can be handled by [approximate] Bayesian inference through an algorithm called ABC, even though there also exist non-Bayesian software resolutions.
5. However, even in such cases, I do not think that Bayesian inference is the only possible resolution. Machine-learning techniques like neural nets, random forests, deep learning, can be classified as frequentist methods since they train on a sample by cross-validation, minimising an error or distance criterion that can be seen as an expectation [under the true model] approximated by a sample average. For instance, Kingman's coalescent model can also be handled by non-Bayesian software resolutions.
6. A final point is that, for point estimation, the Bayesian approach may well produce plug-in estimates. For some loss functions that I called intrinsic losses, the Bayes estimator of the transform $\mathfrak{h}(\theta)$ is the transform $\mathfrak{h}(\hat\theta)$ of the Bayes estimator of $\theta$.