3

An acquaintance of mine has been using this wrong inference formula for years: given

  • a i.i.d. sample $\mathbf{X}={X_1,\dots,X_N}$ for a continuous RV $X$,
  • sample mean $\bar{X}=\frac{\sum X_i}{N}$ and sample standard deviation $\bar{\sigma}=\frac{\sum \left(X_i-\bar{X}\right)^2}{N-1}$

estimate the 0.95-quantile $q_{0.95}$ as

$$q_{0.95} = \bar{X} + 2 \bar{\sigma}$$

(which is not even a decent point estimate - you should at the very least use $q_{0.95} = \bar{X} + 1.645\bar{\sigma}$).

What are the correct confidence intervals for a generic $q$, in the three cases:

  • we know nothing on $X$ (apart from the fact that it's continuous), and we look for an exact (non-asymptotic) answer. I think this answer for the median could be modified for a generic quantile
  • as before, but an asymptotic solution is fine. I guess there should be at least a couple answers here...one for quantiles which aren't close to 0 or 1, and one for quantiles which are. Maybe one based on normal approximation and one based on Poisson?
  • finally, we assume $X$ to have a Gaussian distribution with unknown mean and variance.
DeltaIV
  • 15,894
  • 4
  • 62
  • 104
  • 3
    Your colleague’s formula gives just a point estimate, not an interval. Which are you after? That formula also isn’t necessarily an awful estimator of $q_{0.975}$. I might run some simulations to check out its MSE compared to other estimates. My guess is that it’s pretty good for a normal distribution and then fails for asymmetric and heavy-tailed distributions. – Dave Feb 09 '20 at 12:37
  • 1
    I'll offer my response to [this Cross Validated question](https://stats.stackexchange.com/questions/446802/r-removing-zeros-for-pseudomedian-and-its-confidence-interval-in-wilcox-test/446904#446904), which discusses two approaches for a confidence intervals for a quantile: 1) an approach from Conover based on the binomial distribution; and, 2) confidence intervals by percentile bootstrap. – Sal Mangiafico Feb 09 '20 at 14:21
  • 2
    Confidence intervals for quantiles are commonly known as *tolerance intervals.* – whuber Feb 09 '20 at 15:31
  • @Dave indeed, it's a point estimate not an interval, and that's one of the problems: my acquaintance doesn't take any uncertainty into account, and just plugs that formula in, as if it was a perfect estimate of $q$. The other problem is that "your guess" is that it's not an awful estimator, but I'd rather see some theory or simulations. – DeltaIV Feb 09 '20 at 16:13
  • @whuber good point, I forgot about that: thanks for reminding me. Do you think I should edit the question title? It would surely improve the question quality, but it might make it a bit harder to Google for, and thus potentially less useful. – DeltaIV Feb 09 '20 at 16:15
  • @SalMangiafico sounds cool! I don't think my question is a duplicate, since I don't think the bootstrap can be considered an asymptotic estimate (even though it definitely fails for small $N$). But indeed, at least the first point (distribution-free tolerance interval) is answered by your question. – DeltaIV Feb 09 '20 at 16:18
  • @whuber Wouldn't a tolerance interval be a range where we expect to find, say, 80% of the density, something like $q_{0.1}$ to $q_{0.9}$? – Dave Feb 09 '20 at 17:57
  • @Dave Yes. Tolerance intervals are closely connected to tolerance limits (obviously) and a tolerance limit can be considered to be either a confidence limit for a quantile or an estimate of the quantile (depending on what kind of tolerance limit we're concerned about). – whuber Feb 09 '20 at 18:02
  • @whuber you made me realize that probably what my coworker really needs, it's not a confidence interval for $q_{0.975}$, but rather an estimator such that 95% of times, 95% of the population is _lower_ than this estimator. I _think_ this would be the same than a one-sided confidence interval for $q_{0.975}$, and also the same as a 95%-upper tolerance limit at 95% confidence. Am I right? – DeltaIV Feb 09 '20 at 18:15
  • 1
    I think that is incorrect. Wouldn’t you want to work with $q_{0.95}$ instead of $0.975$? – Dave Feb 09 '20 at 18:20
  • 1
    Yes, the idea looks right. I cannot discern any general relationship between the original $0.975$ value and the $95\%,95\%$ criteria, though. Don't you mean "$0.95$"? – whuber Feb 09 '20 at 18:20
  • @whuber you're right, my mistake. Will fix the question. – DeltaIV Feb 09 '20 at 18:57
  • @Dave indeed it is. Years of thinking in terms of two-sided NHST led me astray. Fixing the question now. – DeltaIV Feb 09 '20 at 18:59
  • @whuber btw, this made me realize that there's another mistake in my coworker's procedure...if he really wanted a decent estimate for $q_{0.95}$, he should have used 1.645 and not 2 in the above formula. – DeltaIV Feb 09 '20 at 19:02

1 Answers1

3
  • The first case was answered in detail in this question.
  • One example of the second case is shown here, where the authors apply a normal approximation to the binomial distribution used in calculations of the first case.

The third case is given by Hahn and Meeker in their handbook Statistical Intervals (2nd ed., Wiley 2017):

A two-sided $100(1-\alpha)\%$ confidence interval for $x_q$, the $q$ quantile of the normal distribution, is

$$ \left[\bar{x}-t_{(1-\alpha/2;\,n-1,\,\delta)}\frac{s}{\sqrt{n}},\;\bar{x}-t_{(\alpha/2;\,n-1,\,\delta)}\frac{s}{\sqrt{n}}\right] $$ where $t_{(\gamma;\,n-1,\,\delta)}$ is the $\gamma$ quantile of a noncentral $t$-distribution with $n-1$ degrees of freedom and noncentrality parameter $\delta = -\sqrt{n}z_{(q)}=\sqrt{n}z_{(1-p)}$.

Here, $z_{(q)}$ denotes $\Phi^{-1}(q)$, the $q$ quantile of the standard normal distribution.

For example, let's assume we drew $n=20$ samples from a normal distribution with unknown mean and standard deviation. The sample mean was $\bar{x}=10.5$ and the sample standard deviation was $s=3.19$. Then, the two-sided $95\%$ confidence interval for the $q=0.25$ quantile $x_{0.25}$ would be given by $(6.42; 9.76)$.

Here is some R code and a small simulation to check the coverage. By changing the parameters, you can run your own simulations:

normquantCI <- function(x, conf_level = 0.95, q = 0.5) {
  
  x <- na.omit(x)
  n <- length(x)
  xbar <- mean(x)
  s <- sd(x)
  ncp <- -sqrt(n)*qnorm(q)
  tval <- qt(c((1 + conf_level)/2, (1 - conf_level)/2), n - 1, ncp)
  se <- s/sqrt(n)
  
  xbar - tval*se
  
}

# Simulate the coverage

set.seed(142857)

q <- 0.25 # Quantile to calculate the CI for
conf_level <- 0.95 # Confidence level
true_mean <- 100 # The true mean of the normal distribution
true_sd <- 15 # True sd of the normal distribution
sampsi <- 20 # The sample size

trueq <- qnorm(q, true_mean, true_sd) # The true quantile

res <- replicate(1e5, {
  citmp <- normquantCI(rnorm(sampsi, true_mean, true_sd), conf_level = conf_level, q = q)
  ifelse(citmp[1] < trueq & citmp[2] > trueq, 1, 0)
})

sum(res)/length(res)
[1] 0.95043
Community
  • 1
COOLSerdash
  • 25,317
  • 8
  • 73
  • 123
  • This is nice and I upvoted (btw, made a minor edit, please check it out). However, I don't think you need to link to an external site for the sake of answering point 2 (the asymptotic case)....I can copy that formula into your answer if you don't feel like doing it yourself. It would also be nice to include the other asymptotic for the other case (when $n$ is large but $p$ is either very close to 0 or 1). I _think_ a Poisson approximation would work, even though probably extreme events theory is the right way to go? – DeltaIV Feb 09 '20 at 16:31
  • @DeltaIV Thanks. I deliberately didn't copy the formula for the normal approximation because I wanted to focus on the third case. I suspect that the formula wouldn't make much sense without more context from the first case. Instead of copy & pasting the formula into my answer, I would encourage you to post it as a separate answer, preferably with a bit more context. You could also develop and present your ideas about the Poisson approximation there. – COOLSerdash Feb 09 '20 at 16:42
  • Rather than having three answers, each focusing on one point, I think it would be better to have **one** answer covering all three points and ideally comparing them (it would also make it easier for me to accept an answer). As a matter of fact, initially the question title was "comparison among confidence [..]", but then it took the "comparison" part away, because I worried that people may choose not to answer, because they wouldn't want to spend time writing a lot of R code. Anyway, if you prefer to focus on the third point only, I'll do as you say & write a separate answer. – DeltaIV Feb 09 '20 at 16:56