11

The simplest definition of a sufficient statistics in the frequentist perspective is given here in Wikipedia. However, I recently came across in a Bayesian book, with the definition $P(\theta|x,t)=P(\theta|t)$. It's stated in the link that both are equivalent, but I don't see how. Also, in that same page, in the section «Other Types of sufficiency» it's stated that both definitions are not equivalent in infinite-dimensional spaces...

Also, how does the predictive sufficiency relates to the classical sufficiency?

An old man in the sea.
  • 5,070
  • 3
  • 23
  • 57

2 Answers2

5

We came across an interesting phenomena a few years ago, when investigating Bayesian model choice with ABC. Which I think is related with this question. There is indeed a notion of sufficiency for Bayesian model choice that does not seem particularly meaningful outside the Bayesian approach.

Given two models $$\mathfrak{M}_1=\{f_\theta(\cdot); \theta\in\Theta\}$$ and $$\mathfrak{M}_2=\{g_\xi(\cdot); \xi\in\Xi\}$$ and a sample $\mathbf{x}=(x_1,\ldots,x_n)$ from one of these two models, a statistic $S$ is sufficient for model choice or across model iff the distribution of $\mathbf{X}$ conditional on $S(\mathbf{X})$ does not depend on either the model index (1 or 2) or the parameter value within the model.

When such sufficient statistics exist, a Bayes factor based on $\mathbf{X}$ is the same as a Bayes factor based on $S(\mathbf{X})$. While this is a definition that is not Bayesian per se, I see no direct application outside Bayesian model choice.

Xi'an
  • 90,397
  • 9
  • 157
  • 575
4

If a statistic $T$ is sufficient in the frequentist way, then $p(\mathbf{x} \mid \theta, t) = p(\mathbf{x} \mid t)$, so \begin{align*} p(\theta \mid \mathbf{x}, t) &= \frac{p(\mathbf{x}\mid t,\theta)p(t \mid \theta) p(\theta)}{p(\mathbf{x}\mid t)p(t)} \\ &= \frac{p(t \mid \theta) p(\theta)}{p(t)} \tag{freq. suff.}\\ &= p(\theta \mid t). \end{align*}

On the other hand, if $T$ is sufficient in the Bayesian way, then \begin{align*} p(\mathbf{x} \mid \theta, t) &= \frac{p(\mathbf{x}, \theta,t)}{p(\theta,t)}\\ &= \frac{p(\theta \mid \mathbf{x},t) p(\mathbf{x},t)}{p(\theta\mid t)p(t)}\\ &= \frac{p(\mathbf{x},t)}{p(t)} \tag{Bayesian suff.}\\ &= p(\mathbf{x} \mid t). \end{align*}

Regarding "predictive sufficiency," what's that?

Edit: If you have Bayesian sufficiency, you have predictive sufficiency: \begin{align*} p(\mathbf{x}' \mid \mathbf{x}) &= \int p(\mathbf{x}' \mid \theta)p(\theta \mid \mathbf{x}) d\theta\\ &= \int p(\mathbf{x}' \mid \theta) p(\theta \mid t)d\theta \tag{Bayesian suff.}\\ &= p(\mathbf{x}' \mid t). \end{align*}

Taylor
  • 18,278
  • 2
  • 31
  • 66