Is there an unbiased estimator of the Hellinger distance between two distributions?

Question

In a setting where one observes $X_1,\ldots,X_n$ distributed from a distribution with density $f$, I wonder if there is an unbiased estimator (based on the $X_i$'s) of the Hellinger distance to another distribution with density $f_0$, namely $$ \mathfrak{H}(f,f_0) = \left\{ 1 - \int_\mathcal{X} \sqrt{f(x)f_0(x)} \text{d}x \right\}^{1/2}\,. $$

Answer 1

I don't know how to construct (if it exists) an unbiased estimator of the Hellinger distance. It seems possible to construct a consistent estimator. We have some fixed known density $f_0$, and a random sample $X_1,\dots,X_n$ from a density $f>0$. We want to estimate $$ H(f,f_0) = \sqrt{1 - \int_\mathscr{X} \sqrt{f(x)f_0(x)}\,dx} = \sqrt{1 - \int_\mathscr{X} \sqrt{\frac{f_0(x)}{f(x)}}\;\;f(x)\,dx} $$ $$ = \sqrt{1 - \mathbb{E}\left[\sqrt{\frac{f_0(X)}{f(X)}}\;\;\right] }\, , $$ where $X\sim f$. By the SLLN, we know that $$ \sqrt{1 - \frac{1}{n} \sum_{i=1}^n \sqrt{\frac{f_0(X_i)}{f(X_i)}}} \quad \rightarrow H(f,f_0) \, , $$ almost surely, as $n\to\infty$. Hence, a resonable way to estimate $H(f,f_0)$ would be to take some density estimator $\hat{f_n}$ (such as a traditional kernel density estimator) of $f$, and compute $$ \hat{H}=\sqrt{1 - \frac{1}{n} \sum_{i=1}^n \sqrt{\frac{f_0(X_i)}{\hat{f_n}(X_i)}}} \, . $$

Answer 2

No unbiased estimator either of $\mathfrak{H}$ or of $\mathfrak{H}^2$ exists for $f$ from any reasonably broad nonparametric class of distributions.

We can show this with the beautifully simple argument of

Bickel and Lehmann (1969). Unbiased estimation in convex families. The Annals of Mathematical Statistics, 40 (5) 1523–1535. (project euclid)

Fix some distributions $F_0$, $F$, and $G$, with corresponding densities $f_0$, $f$, and $g$. Let $H(F)$ denote $\mathfrak{H}(f, f_0)$, and let $\hat H(\mathbf X)$ be some estimator of $H(F)$ based on $n$ iid samples $X_i \sim F$.

Suppose that $\hat H$ is unbiased for samples from any distribution of the form $$M_\alpha := \alpha F + (1 - \alpha) G .$$ But then \begin{align} Q(\alpha) &= H(M_\alpha) \\&= \int_{x_1} \cdots \int_{x_n} \hat H(\mathbf X) \,\mathrm{d}M_\alpha(x_1) \cdots\mathrm{d}M_\alpha(x_n) \\&= \int_{x_1} \cdots \int_{x_n} \hat H(\mathbf X) \left[ \alpha \mathrm{d}F(x_1) + (1-\alpha) \mathrm{d}G(x_1) \right] \cdots \left[ \alpha \mathrm{d}F(x_n) + (1-\alpha) \mathrm{d}G(x_n) \right] \\&= \alpha^n \operatorname{\mathbb{E}}_{\mathbf X \sim F^n}[ \hat H(\mathbf X)] + \dots + (1 - \alpha)^n \operatorname{\mathbb{E}}_{\mathbf X \sim G^n}[ \hat H(\mathbf X)] ,\end{align} so that $Q(\alpha)$ must be a polynomial in $\alpha$ of degree at most $n$.

Now, let's specialize to a reasonable case and show that the corresponding $Q$ is not polynomial.

Let $F_0$ be some distribution which has constant density on $[-1, 1]$: $f_0(x) = c$ for all $\lvert x \rvert \le 1$. (Its behavior outside that range doesn't matter.) Let $F$ be some distribution supported only on $[-1, 0]$, and $G$ some distribution supported only on $[0, 1]$.

Now \begin{align} Q(\alpha) &= \mathfrak{H}(m_\alpha, f_0) \\&= \sqrt{1 - \int_{\mathbb R} \sqrt{m_\alpha(x) f_0(x)} \mathrm{d}x} \\&= \sqrt{1 - \int_{-1}^0 \sqrt{c \, \alpha f(x)} \mathrm{d}x - \int_{0}^1 \sqrt{c \, (1 - \alpha) g(x)} \mathrm{d}x} \\&= \sqrt{1 - \sqrt{\alpha} B_F - \sqrt{1 - \alpha} B_G} ,\end{align} where $B_F := \int_{\mathbb R} \sqrt{f(x) f_0(x)} \mathrm{d}x$ and likewise for $B_G$. Note that $B_F > 0$, $B_G > 0$ for any distributions $F$, $G$ which have a density.

$\sqrt{1 - \sqrt{\alpha} B_F - \sqrt{1 - \alpha} B_G}$ is not a polynomial of any finite degree. Thus, no estimator $\hat H$ can be unbiased for $\mathfrak{H}$ on all of the distributions $M_\alpha$ with finitely many samples.

Likewise, because $1 - \sqrt{\alpha} B_F - \sqrt{1 - \alpha} B_G$ is also not a polynomial, there is no estimator for $\mathfrak{H}^2$ which is unbiased on all of the distributions $M_\alpha$ with finitely many samples.

This excludes pretty much all reasonable nonparametric classes of distributions, except for those with densities bounded below (an assumption nonparametric analyses sometimes make). You could probably kill those classes too with a similar argument by just making the densities constant or something.