8

While doing some simulations, I realised that the sample quantile is a biased estimator of the true quantile. And, according to my simulations, a potentially very biased one.

I was surprised with that result since the empirical CDF isn't biased, but after some internet research, I discovered it was true.

I tried to figure out where that bias comes from, but working with sample quantiles is quite difficult. Does anyone have a demonstration of that bias (and, ideally, a quantification)?

Thomas
  • 83
  • 3
  • 2
    The ECDF is unbiased for the cdf but how would you get from the ECDF to a sample quantile? – Glen_b Nov 12 '13 at 00:13
  • 3
    There is no such thing as "the sample quantile". There are many definitions of sample quantiles. You need to specify which one you mean. – Rob Hyndman Nov 12 '13 at 12:01

3 Answers3

4

Bias in estimating $p$-quantiles is investigated in a distribution-free way in

http://www.sciencedirect.com/science/article/pii/S016771520000242X

(a pdf can be found on the same page). The authors focus on the quantile estimator based on ECDF inversion. No assumptions on the underlying distribution is made (except finite second moment), thus also discrete distributions are included.

Some highlights:

  • Bias is proportional to the standard deviation $\sigma$ of the underlying distribution

  • Bias is smaller in central quantiles than in extreme ones. This stems from the fact that among all distributions with standard deviation $\sigma < \infty$, bias oscillates in an interval of length $\frac{\sigma}{\sqrt{p (1-p)}}$. Strikingly, this does not depend on the sample size $n$.

  • For $np>3$, among all standardized distributions (mean 0, standard deviation 1), the worst bias is associated with the distribution having an atom of probability $p$ at $-\sqrt{(1-p)/p}$ and an atom of probability $1-p$ at $\sqrt{p/(1-p)}$.

Michael M
  • 10,553
  • 5
  • 27
  • 43
1

Just to add to this old post, The ECDF is only unbiased at high samples sizes. At low values of N it is biased. Take the trivial case of N=1 and the ECDF takes a value of 1 at and above the sample value. Ask yourself what is the value of the underlying distribution that gives a probability of 1?

The bias actually exceeds sqrt(2*pi)/(2N)*SD or 1.25/N * SD so for an N of 5 this is a 0.25 SD bias.

Instead of an ECDF based on k/N, try (k-0.5)/N to get an unbiased ECDF. That might give you unbiased sample quantiles. It also ensures that ECDF(x)=1-ECDF(-x) which is enjoyed by all other cumulative distributions.

In my very humble opinion the ECDF as defined and used is a huge misnomer. It biases Kolmogorov Smirnov, Lilliefors and other standard tests at low N.

Check out Gilchrist "Statistical modelling with Quantile Functions"

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
user2092957
  • 111
  • 1
  • 1
    This is an interesting point, but technically, the ECDF **is** unbiased! You refer to the fact that, say, after seeing that ECDF(x)=1, you know the error can only have one sign, so you have a conditional bias of sort. But the frequentist property of unbiasedness refer to the situation before seeing any data, not the conditional bias you refer to. – kjetil b halvorsen Oct 19 '16 at 20:37
0

There exists a unique true sample quantile definition (which is not the one usually presented). See: http://dx.doi.org/10.1155/2014/326579

user153836
  • 9
  • The article is interesting but many readers would benefit from a summary of the arguments and why the many existing definitions are mis-guided. – mdewey Mar 20 '17 at 16:28