3

I am trying to understand the connection between Bootstrap and Maximum Likelihood Estimator. Is the connection of the following format?

Let $ \mathcal{F}_{\theta} $ be a parametrized family of distributions. Let us consider a distribution $ \mathcal{F}_{\theta_0} $ with unknown value of $ \theta_0 $. Let $ \vec{X} $ be an iid sample of size N from distribution $ \mathcal{F}_{\theta_0} $. And let $ \theta^* $ the MLE estimate of $ \theta $ under the above assumptions, that is

\begin{align*} \theta^* = \arg\max_{\theta} \prod_{x \in \vec{X}} P_{\mathcal{F}_{\theta}}(x) \ . \end{align*}

We are concerned about the variance of a statistics $ S(\vec{X}) $ over the sampled data $ \vec{X} $. Let $ var_B S(\vec{X}) $ be a bootstrapped variance of statistics $ S $ computed over B rounds, that is, we sample a sample $\vec{X}_i$ with replacement from $ \vec{X} $ and compute variance of $ \{S(\vec{X}_i), i=1,\dots,B \} $.

Then \begin{align*} \lim _{B\to \infty} [var_B S(\vec{X})] = var_{x \sim \mathcal{F}_{\theta^*}} S(x) \ . \end{align*} Is the limit correct? Does N have to go to infinity as well?

(I would appreciate answers beyond simple references).

Edit: Note that in the right side of the equation, $x$ is sampled from $\mathcal{F}_{\theta^*}$, not $\mathcal{F}_{\theta_0}$.

ashim
  • 415
  • 5
  • 10

1 Answers1

3

Ok, so after the edit it's clearer that the question is different from what I thought. The answer is still 'No'.

Suppose $X$ has a Cauchy distribution and take $S(X)=\bar X$. $\mathrm{var}_B[\bar X]$ is always finite, and has a finite limit as $B\to\infty$ for fixed $N$, but $\mathrm{var}_{X\sim\textrm{Cauchy}(\theta^*)}[\bar X]$ is infinite.

Less extreme, take $X\sim N(\mu,1)$. Then $\mathrm{var}_{X\sim P_{\theta^*}}[\bar X]=1/N$, but $$\lim_{B\to\infty}\mathrm{var}_{B}[\bar X]=\hat\sigma^2/N$$ where $\hat\sigma^2$ is the sample variance in that sample.

In the second case the equality straightforwardly holds for the limit as $N\to\infty$. In the first case it holds in the sense that the bootstrap variance increases a.s. without bound as $N\to\infty$.

Thomas Lumley
  • 21,784
  • 1
  • 22
  • 73
  • Thank you for your answer. But the MLE estimator for sample $\overrightarrow{X}$ is $\mu = \bar{X}$ where, I assume that, by $\bar{X}$ you meant an average among 0 and 1 values from the Bernoulli sampled $\overrightarrow{X}$. For your example, the original limit equation holds. Am I missing something? – ashim Jul 12 '21 at 01:24
  • A question on the 2nd example. Shouldn't the information that the variance equals 1 be used in the lim→∞var[¯] case as well? – ashim Jul 12 '21 at 22:05
  • Not if what you want is the bootstrap variance. You could set up a resampling estimator that made the same assumptions as the MLE, and then it would give the same answer as the MLE (eg, a parametric bootstrap), but an ordinary bootstrap doesn't – Thomas Lumley Jul 13 '21 at 00:00