8

If you fit a quantile regression for the 5th and 95th percentile this is often described as an estimate of a 90% prediction interval. This is the most prevalent it seems in the machine learning domain where random forests has been adapted to predict the quantiles of each leaf node or GBM with a quantile loss function.

Is this best characterized as a confidence or prediction interval and why?

AdamO
  • 52,330
  • 5
  • 104
  • 209
B_Miner
  • 7,560
  • 20
  • 81
  • 144

2 Answers2

12

Definitely a prediction interval, see for example here.

Quantile regression for the $5^\textrm{th}$ and $95^\textrm{th}$ quantiles attempts to find bounds $y_0({\bf x})$ and $y_1({\bf x})$, on the response variable $y$ given predictor variables ${\bf x}$, such that $$ \mathbb{P}\left(Y\le y_0({\bf X})\right)=0.05 \\ \mathbb{P}\left(Y\le y_1({\bf X})\right)=0.95 $$ so $$ \mathbb{P}\left(\,y_0({\bf X})\le Y\le y_1({\bf X})\,\right)\ =\ 0.90 $$ which is by definition a $90\%$ prediction interval.

A $90\%$ prediction interval should contain (as-yet-unseen) new data $90\%$ of the time. In contrast, a $90\%$ confidence interval for some parameter (e.g. the mean) should contain the true mean unless we were unlucky to the tune of 1-in-10 in the data used to construct the interval.

Creosote
  • 2,102
  • 8
  • 11
-3

After posting this I realized it is most accurately called a confidence interval, regardless of the terminology used by sci-kit learn.

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3885826/

"However, it must be kept in mind that the resulting confidence intervals are a model approximation rather than true statistics".

B_Miner
  • 7,560
  • 20
  • 81
  • 144
  • 9
    Most "accurately" called? No. Most commonly? Probably. Take a look at the documentation for the *‘randomUniformForest* **R** package, which has a "confInt" type for the *predict* method which yields, according to the manual pages, *prediction intervals*. So there appears to be widespread confusion between the two concepts (see also the reference I gave in my main answer). I'm going to stick to my guns -- the correct terminology is **prediction interval** for **data** and confidence interval for a parameter -- but it looks like very few people will care if you get it wrong. – Creosote Sep 09 '15 at 20:12
  • 2
    Sorry but that is not correct. Confidence intervals are intervals around expected values such as E(y|X) while prediction intervals are intervals around y|X. See this http://stats.stackexchange.com/questions/16493/difference-between-confidence-intervals-and-prediction-intervals – B_Miner Sep 09 '15 at 23:51
  • 2
    Yes, CI's are for E(Y|x), PI's for Y|x, but in the Meinshausen paper it is the latter case that we're looking at. Quote: "Prediction Intervals How reliable is a prediction for a new instance?". The new instance is Y, given explanatory variables x. The author goes on: "a new observation of Y, for X=x, is with high probability in the interval I(x)". This is a statement about Y itself, not the mean of all possible Y, given x. If you are asking in your original question if Meinshausen is using the wrong terminology then I remain quite sure that he is not. – Creosote Sep 10 '15 at 06:45
  • I appreciate where you are coming from B_miner, I had the same "second thoughts" about this but I think the easiest thing to consider is that quantile regression (irrelevant of the algorithm for a moment) is a different approach to constructing prediction intervals as it indeed estimates conditional quantile functions from the available data. If we fit the conditional quantile function at the 2.5% and 97.5% levels we can obtain prediction intervals for "nominal" 95% coverage from the corresponding interval. – usεr11852 Jul 11 '20 at 04:00