0

Let $f(x)$ be some smooth univariate density, and let the leave-one-out Nadaraya-Watson estimator $\widehat{f}_{-i}(x)$ be defined as follows: $\widehat{f}_{-i}(x)=\frac{1}{(n-1)h}\sum_{j=1,j\neq i}^nK(\frac{X_j-x}{h})$, where $K(\cdot)$ is the kernel function and bandwidth $h\rightarrow 0$ at some specified speed so that we have $\underset{x\in J}{\sup} |\widehat{f}_{-i}(x)-f(x)|=o_{P}(n^{-1/4})$, where $J$ is a compact subset of the support of $X$ that excludes the boundary of the support. (An example for the unknown true density $f(x)$ could be the standard normal density, and an example for the known interval $J$ would be $J=[-50,50]$ )

I have the following two statements:

$\frac{1}{n}\sum_{i=1}^n|\mathbf{1}(X_i\in J)(\widehat{f}_{-i}(X_i)-f(X_i))|=o_{p}(n^{-1/2})$

$|\frac{1}{n}\sum_{i=1}^n \widehat{A}(X_i)\mathbf{1}(X_i\in J)(\widehat{f}_{-i}(X_i)-f(X_i))|=o_{p}(n^{-1/2})$, where $\widehat{A}(x)$ is a consistent estimator of some function $A(x)$, and is uniformly bounded on the support of $X$ with probability 1, and $\mathbf{1}(\cdot)$ is the indicator function.

Which of these two statements is valid or more likely to be valid? You can add additional assumptions if needed. Intuitive or rigorous justification, related reference are all welcome, thanks!

T34driver
  • 1,608
  • 5
  • 11

1 Answers1

2

The first statement is much stronger, because the absolute value is inside the sum. The impact of including $\hat A()$ will depend on what it is. If it's deterministic or consistently estimated, it shouldn't make much difference, but if you took $$\hat A(X_i)= \mathrm{sign}(\hat f_{-i}(X_i)-f(X_i))$$ you'd be effectively moving the absolute value inside the sum. I'm assuming $J$ is an arbitrary compact subset of the support of $X$, otherwise you could take $J$ to be a single point (or empty).

I would say the first statement is unlikely to be true, since I would not expect the individual terms to be $o_p(n^{-1/2})$ (or even $O_p(n^{-1/2})$, and there is no cancellation.

The second statement is more plausible; it would easily be true if $(\hat f_{-i}(X_i)-f(X_i))$ were independent and mean zero. You have bounds on the maximum bias, but you'd need to also control the proportion of observations $X_i$ in regions where the bias was near the maximum. If $f$ had a pointy mode (where $\hat f$ will be biased down) or there were regions where the density wasn't bounded away from zero (so $\hat f_{-i}(X_i)$ will be biased down) then you can't rely on the bias cancelling (because you can choose $J$ to cover just those regions).

Thomas Lumley
  • 21,784
  • 1
  • 22
  • 73
  • Thanks a lot! These comments are very helpful. I agree with your intuition that the first statement is unlikely to be true. And your technical comments regarding $\widehat{A}()$, $f$ and $J$ are quite shrewd too. To assure you, $\widehat{A}()$ is consistently estimated, $J$ is the compact support that only exclude tiny intervals on the boundary, $f$ is smooth and positive everywhere on $J$. Will update these in the question too. Given these conditions, can you elaborate a bit more on the steps needed to establish the $o_{p}(n^{-1/2})$ in the second statement? – T34driver Sep 17 '20 at 23:28
  • For example, can you elaborate a bit more on the following three points: (1). if I understood it correctly, you were suggesting making use of $max_{i}\{E(\widehat{f}_{-i}(X_i)|X_i)-f(X_i)\}=O_{p}(?)$. What is this bound(or how to get this bound) in terms of bandwidth? (2). How to control "the proportion of observations in the regions where the bias was near the maximum"? (3). How does (1) and (2) contributes to showing $o_{p}(n^{-1/2})$? Thanks. – T34driver Sep 17 '20 at 23:40
  • You gave the bound on the supremum of the error as $o_p(n^{-1/4})$. The maximum bias will be of that order, and doesn't necessarily average out to anything smaller, so the question is how many of the $n$ observations have bias near $o_p(n^{-1/4})$. I don't know how you'd control that, but it looks like you need to. Maybe you could make the mysterious $\hat A$ small where the bias is large, if you control it? – Thomas Lumley Sep 18 '20 at 01:29
  • Thanks! I cannot make the mysterious $\widehat{A}$ behave like that unfortunately :) All we know about $\widehat{A}$ is that it's bounded. But I guess what you mentioned is already controlled, right? Because on $J$ the order of the bias is uniformly controlled, and when we get outside $J$(go near the boundary) where bias is supposed to be large, the indicator function will be zero. – T34driver Sep 18 '20 at 04:15
  • It's not clear whether the bias is sufficiently controlled on $J$. If the bias were of size. say, $n^{-1/3}$, everywhere, the average bias would also be $n^{-1/3}$, which is is not $o(n^{-1/4})$. The bias needs to be large only at a small fraction of points. – Thomas Lumley Sep 18 '20 at 06:17
  • Thanks! I guess I have shown the second statement without the $\widehat{A}$ can achieve at most $O_{p}(n^{-1/2})$. – T34driver Sep 18 '20 at 16:48