15

I have a couple of questions for prediction and tolerance intervals.

Let's agree on the definition of the tolerance intervals first: We are given a confidence level, say 90%, the percentage of the population to capture, say 99%, and a sample size, say 20. The probability distribution is known, say normal for convenience. Now, given the above three numbers (90%, 99% and 20) and the fact that the underlying distribution is normal, we can compute the tolerance number $k$. Given a sample $(x_1,x_2,\ldots,x_{20})$ with mean $\bar{x}$ and standard deviation $s$, the tolerance interval is $\bar{x}\pm ks$. If this tolerance interval captures 99% of the population, then the sample $(x_1,x_2,\ldots,x_{20})$ is called a success and the requirement is that 90% of the samples are successes.

Comment: 90% is the a priori probability for a sample to be a success. 99% is the conditional probability that a future observation will be in the tolerance interval, given that the sample is a success.

My questions: Can we see prediction intervals as tolerance intervals? Looking on the web I got conflicting answers on this, not to mention that nobody really defined the prediction intervals carefully. So, if you have a precise definition of the prediction interval (or a reference), I would appreciate it.

What I understood is that a 99% prediction interval for instance, does not capture 99% of all future values for all samples. This would be the same as a tolerance interval that captures 99% of the population with 100% probability.

In the definitions I found for a 90% prediction interval, 90% is the a priori probability given a sample, say $(x_1,x_2,\ldots,x_{20})$ (size is fixed) and a single future observation $y$, that $y$ will be in the prediction interval. So, it seems that both the sample and the future value are both given at the same time, in contrast to the tolerance interval, where the sample is given and with a certain probability it is a success, and under the condition that the sample is a success, a future value is given and with a certain probability falls into the tolerance interval. I am not sure if the above definition of the prediction interval is right or not, but it seems counterintuitive (at least).

Any help?

whuber
  • 281,159
  • 54
  • 637
  • 1,101
Ioannis Souldatos
  • 253
  • 2
  • 5
  • 4
    One-sided tolerance intervals for a normal sampling might help the understanding of this notion. An upper $99\%$-tolerance bound is nothing but an upper confidence bound of the $99\%$-quantile of the assumed distribution of the model. Therefore in case of a normal distribution this is an upper confidence bound of the parameter $\mu + k\sigma$ where $k=z_{99\%}$ is $99\%$ of the standard gaussian distribution. – Stéphane Laurent Apr 18 '12 at 17:41
  • This is a good reformulation, Stéphane, because it immediately shows there are several kinds of tolerance limits: one can ask for an *upper* confidence limit on $\mu + z_{0.99}\sigma$, for a *lower* confidence limit on $\mu + z_{0.99}\sigma$, or for (say) an unbiased *estimate* of that parameter. All three are called "tolerance limits" in the literature. – whuber Apr 18 '12 at 18:34
  • I think you rather wanted to say a lower confidence limit on $\mu - z_{0.99}\sigma$ ? – Stéphane Laurent Apr 18 '12 at 18:43
  • 2
    Actually, no, Stéphane (which is why I took care to repeat the formula for the parameter). There are also three similar definitions for a *lower* tolerance limit. E.g., we might want to *under*-estimate the upper 99th percentile of the population, but to control the amount of underestimation we insist there be (say) a 5% chance that our underestimate will still be too high. This will allow us to say things like "The data show, with 95% confidence, that the 99th percentile of the population exceeds such-and-such a value." – whuber Apr 18 '12 at 18:47

2 Answers2

16

Your definitions appear to be correct.

The book to consult about these matters is Statistical Intervals (Gerald Hahn & William Meeker), 1991. I quote:

A prediction interval for a single future observation is an interval that will, with a specified degree of confidence, contain the next (or some other prespecified) randomly selected observation from a population.

[A] tolerance interval is an interval that one can claim to contain at least a specified proportion, p, of the population with a specified degree of confidence, $100(1-\alpha)\%$.

Here are restatements in standard mathematical terminology. Let the data $\mathbf{x}=(x_1,\ldots,x_n)$ be considered a realization of independent random variables $\mathbf{X}=(X_1,\ldots,X_n)$ with common cumulative distribution function $F_\theta$. ($\theta$ appears as a reminder that $F$ may be unknown but is assumed to lie in a given set of distributions ${F_\theta \vert \theta \in \Theta}$). Let $X_0$ be another random variable with the same distribution $F_\theta$ and independent of the first $n$ variables.

  1. A prediction interval (for a single future observation), given by endpoints $[l(\mathbf{x}), u(\mathbf{x})]$, has the defining property that

    $$ \inf_\theta\{{\Pr}_\theta(X_0 \in [l(\mathbf{X}), u(\mathbf{X})])\}= 100(1-\alpha)\%.$$

    Specifically, ${\Pr}_\theta$ refers to the $n+1$ variate distribution of $(X_0, X_1, \ldots, X_n)$ determined by the law $F_\theta$. Note the absence of any conditional probabilities: this is a full joint probability. Note, too, the absence of any reference to a temporal sequence: $X_0$ very well may be observed in time before the other values. It does not matter.

    I'm not sure which aspect(s) of this may be "counterintuitive." If we conceive of selecting a statistical procedure as an activity to be pursued before collecting data, then this is a natural and reasonable formulation of a planned two-step process, because both the data ($X_i, i=1,\ldots,n$) and the "future value" $X_0$ need to be modeled as random.

  2. A tolerance interval, given by endpoints $(L(\mathbf{x}), U(\mathbf{x})]$, has the defining property that

    $$ \inf_\theta\{{\Pr}_\theta\left(F_\theta(U(\mathbf{X})) - F_\theta(L(\mathbf{X})\right) \ge p)\} = 100(1-\alpha)\%.$$

    Note the absence of any reference to $X_0$: it plays no role.

When $\{F_\theta\}$ is the set of Normal distributions, there exist prediction intervals of the form

$$l(\mathbf{x}) = \bar{x} - k(\alpha, n) s, \quad u(\mathbf{x}) = \bar{x} + k(\alpha, n) s$$

($\bar{x}$ is the sample mean and $s$ is the sample standard deviation). Values of the function $k$, which Hahn & Meeker tabulate, do not depend on the data $\mathbf{x}$. There are other prediction interval procedures, even in the Normal case: these are not the only ones.

Similarly, there exist tolerance intervals of the form

$$L(\mathbf{x}) = \bar{x} - K(\alpha, n, p) s, \quad U(\mathbf{x}) = \bar{x} + K(\alpha, n, p) s.$$

There are other tolerance interval procedures: these are not the only ones.

Noting the similarity among these pairs of formulas, we may solve the equation

$$k(\alpha, n) = K(\alpha', n, p).$$

This allows one to reinterpret a prediction interval as a tolerance interval (in many different possible ways by varying $\alpha'$ and $p$) or to reinterpret a tolerance interval as a prediction interval (only now $\alpha$ usually is uniquely determined by $\alpha'$ and $p$). This may be one origin of the confusion.

whuber
  • 281,159
  • 54
  • 637
  • 1,101
  • 2
    The confusion among these intervals is real. A decade ago I had several difficult conversations with a government statistician who was ignorant of the difference and (virulently) unable to recognize there is one. Her prominent role in creating guidance, reviewing reports, advising case workers, distributing software, and even peer-reviewed publication has promoted the continuance of these misconceptions. So beware! – whuber Apr 18 '12 at 18:25
  • Very nice answer, thanks. I had heart some statisticians saying that a prediction interval is a tolerance interval with $p=50\%$. Is there a real fact behind this idea ? In other words, is it true that $k(\alpha,n)=K(\alpha,n,0.5)$, or something like that ? – Stéphane Laurent Apr 18 '12 at 19:00
  • 3
    No, that's not true @Stéphane. To see why not, consider the case of extremely large $n$ and moderate confidence, say 95%. With $p=50\%$, the two-sided tolerance interval should therefore be extremely close to some middle 50% of the distribution, so by definition there's only 50% chance that $X_0$ will lie within it, not the desired 95%. That's a huge difference! Intuitively, a tolerance interval for 95% of the population should be sort of close to a prediction interval with 95% confidence, but they still don't exactly agree. – whuber Apr 18 '12 at 19:12
  • I have just thought about this and I believe the fact is the following : $\boxed{k(\alpha,n) \approx K(50\%,n,1-\alpha)}$ when $n$ is large. This is easy to see when $K$ is the classical tolerance factor given with the help of the non-central t distribution (the $50\%$-quantile is the non-centrality parameter $z_{1-\alpha}/\sqrt{n}$) – Stéphane Laurent Apr 18 '12 at 19:32
  • @whuber. Thank you for the answer. I will have to make sure I understand it, before I mark it correct. Give me some time to "digest" it. – Ioannis Souldatos Apr 18 '12 at 21:18
  • I see where you're coming from, Stéphane. It would actually be more accurate to use $K(0\%, n, 1-\alpha)$: the confidence interval then reduces to an estimate of the quantile(s) (tantamount to the third kind of tolerance interval I described). But this is purely an asymptotic result. Its accuracy relies on the sample of $n$ values giving an accurate estimate of the quantiles; for this purpose, $n$ has to be *much* larger than for accurately estimating a mean (because an estimated SD is involved). – whuber Apr 18 '12 at 21:46
  • @whuber Consider the one-sided case. The median of the probability to fall below the $(\alpha=50\%, p=100p\%)$-upper tolerance bound is $p$, whereas the mean of the probability to fall below the $100p\%$-upper prediction bound is $p$. – Stéphane Laurent Nov 18 '12 at 13:34
  • @whuber See my animated pdf http://cjoint.com/data/0KsoNL6kwv2.htm – Stéphane Laurent Nov 18 '12 at 13:40
  • @StéphaneLaurent I know this is old, but it seems like you eventually concluded that a 95% prediction interval is approximately equal to a tolerance interval with 95% coverage and 50% confidence, right? If so, do you still agree with that? That's what I've thought to be the case, but more through intuition and simulation than any mathematical reasoning. Simulating it shows they are not exactly equal, but fairly close even at lower N (~100). – MichiganWater Jul 29 '20 at 07:46
  • @Michigan Be aware that this is a very special case of a prediction interval: namely, the prediction of a single additional independent observation. Many other prediction intervals are in use, such as predicting the largest of $m$ new observations (or their mean, etc.). – whuber Jul 29 '20 at 13:29
  • @whuber Interesting. I've never seen "prediction interval" used to describe anything other than the prediction of an additional observation. Good for me to be aware. Thanks for pointing that out. – MichiganWater Jul 30 '20 at 17:44
  • @Michigan Such intervals are in fairly common use in environmental pollution monitoring systems. I suspect they might be used in industrial quality control settings, too, but I'm not as conversant with that literature. – whuber Jul 30 '20 at 17:49
  • Refering to the MichiganWater's question which seems to be answered confirmative by @whuber. I also try to develop intuition about tolerance intervals and is it justified to think of prediction intervals as of kind of "point estimate" and tolerance intervals as of "interval estimate" of the same quantity, i.e. of the interval containing e.g. 95% of the population? – Wassermann Sep 27 '20 at 14:34
  • @Wassermann You may think of such a tolerance interval as being a point estimate of a quantile, for which you may construct a confidence interval if you like; but conceptually a prediction interval differs because its target is a random variable rather than a property of the underlying distribution. In particular, a prediction interval is always an interval: except with discrete distributions, it's useless to make any kind of specific prediction of another random quantity, because usually you will be correct with zero probability. – whuber Sep 27 '20 at 15:01
  • @whuber thanks for an answer! It seems I still have to give TI some more thought :\ – Wassermann Sep 28 '20 at 20:13
1

As I understand things, for normal tolerance limits, the value of $K(\alpha,p)$ comes from a non central t percentile. Clearly, to W Huber's point, there are some statisticians who are unfamiliar with the idea of tolerance limits versus prediction limits; the idea of tolerance seems to arise mostly in engineering design and manufacturing, as opposed to clinical biostatistics. Perhaps the reason for lack of familiarity with tolerance intervals, and the confusion with prediction intervals, is the context in which one receives his or her statistical training.

Firebug
  • 15,262
  • 5
  • 60
  • 127
Scott P.
  • 11
  • 1