13

I was reading a blog post by the statistician William Briggs, and the following claim interested me to say the least.

What do you make of it?

What is a confidence interval? It is an equation, of course, that will provide you an interval for your data. It is meant to provide a measure of the uncertainty of a parameter estimate. Now, strictly according to frequentist theory—which we can even assume is true—the only thing you can say about the CI you have in hand is that the true value of the parameter lies within it or that it does not. This is a tautology, therefore it is always true. Thus, the CI provides no measure of uncertainty at all: in fact, it is a useless exercise to compute one.

Link: http://wmbriggs.com/post/3169/

Five σ
  • 453
  • 1
  • 3
  • 11
  • 3
    Without a precise reference, there is, most crucially, no context here. There is also no way to get indications of the style and credentials of William Briggs (not known to me). It could be that here is someone who just likes to be provocative and outrageous. There are, naturally, deep and difficult technical and philosophical issues here too, which are the question, but asking us to debate a quotation with no background is (one view only) unlikely to be fruitful. – Nick Cox Feb 20 '15 at 16:21
  • 1
    @NickCox With regards to the omission of relevant context, I have now edited the initial post. – Five σ Feb 20 '15 at 16:22
  • 6
    Thanks very much for providing the back-up. It's only a comment and I lack the inclination to extend it, but my three-word reaction is that the last sentence is **an exaggerated claim**. You can hope for much fuller answers. – Nick Cox Feb 20 '15 at 16:28
  • 1
    @NickCox No problem Nick. However, I do appreciate your sentiments as it was sloppy of me to not reference my question. – Five σ Feb 20 '15 at 16:28
  • 2
    @Nick I would say Briggs succeeded in one of his two objectives: "Today’s thoughts are just a sketch to help clear my mind and begin a discussion. Meaning, it’s likely I will have fallen prey to my own complaint" (that your "neighborhood statistician" is a "sloppy thinker"). – whuber Feb 20 '15 at 17:46

2 Answers2

25

He's referring, rather clumsily, to the well known fact that frequentist analysis doesn't model the state of our knowledge about an unknown parameter with a probability distribution, so having calculated a (say 95%) confidence interval (say 1.2 to 3.4) for a population parameter (say the mean of a Gaussian distribution) from some data you can't then go ahead & claim that there's a 95% probability of the mean falling between 1.2 and 3.4. The probability's one or zero—you don't know which. But what you can say, in general, is that your procedure for calculating 95% confidence intervals is one that ensures they contain the true parameter value 95% of the time. This seems reason enough for saying that CIs reflect uncertainty. As Sir David Cox put it†

We define procedures for assessing evidence that are calibrated by how they would perform were they used repeatedly. In that sense they do not differ from other measuring instruments.

See here & here for further explanation.

Other things you can say vary according to the particular method you used to calculate the confidence interval; if you ensure the values inside have greater likelihood, given the data, than the points outside, then you can say that (& it's often approximately true for commonly used methods). See here for more.

† Cox (2006), Principles of Statistical Inference, §1.5.2

Community
  • 1
Scortchi - Reinstate Monica
  • 27,560
  • 8
  • 81
  • 248
  • 1
    That's Sir David Cox, I imagine. – Nick Cox Feb 20 '15 at 18:22
  • @NickCox: It is indeed. – Scortchi - Reinstate Monica Feb 21 '15 at 21:44
  • Is Sir David's quoted analogy correct? (Not a correct quote, but a correct analogy.) I don't imagine a thermometer which 95% of the time reports the temperature $\pm\epsilon$, but 5% of the time reports temperature outside of $\pm\epsilon$ -- and perhaps far outside of that range? – Wayne Feb 21 '15 at 22:22
  • @Wayne: Not quite sure what you're getting at. Perhaps I should say that the quote's about frequentist inference in general, rather than confidence intervals in particular. Your unusual thermometer seems well characterized as a measuring instrument by the description you've given of how it would perform were it used repeatedly. Of course a *single* confidence interval doesn't summarize *everything* about the uncertainty involved in estimating a parameter. – Scortchi - Reinstate Monica Feb 21 '15 at 22:43
  • @Scortchi: I was just trying to make an analogy, based on Sir David's remark, that felt accurate and had enough detail to be, well, quirky and something I wouldn't buy if it existed. To the extent that it is actually analogous to a CI, it makes me want to not take CI's very seriously. – Wayne Feb 22 '15 at 02:24
  • I don't understand the difference between saying "there's a 95% probability of the mean falling between 1.2 and 3.4" and saying "the CI calculation that I use results in an interval that, in 95% of all cases, contains the mean". I get that the argument here is that the true value IS or IS NOT in that interval, and your calculation doesn't change that - but in the first statement, nobody is saying that the CI calculation affects the probability?! I'm not sure why it would be wrong to go ahead and claim that there's a 95% probability that the mean falls within my CI. – PoorYorick Jun 18 '19 at 12:12
  • 1
    @Spectrosaurus: The posts I linked to go into this in more detail. In fine, the population mean $\mu$ isn't modelled as a random variable; the data $\vec{X}_\mu$ are, with a distribution that depends on $\mu$, & the confidence interval $(b_\mathrm{L}(X_\mu), b_\mathrm{U}(X_\mu))$ is a function of the data. $\Pr[b_\mathrm{U}(\vec{X}_\mu) < \mu < b_\mathrm{U}(\vec{X}_\mu)]=0.95$ defines a valid confidence interval with 95% coverage, whatever value $\mu$ might have. So if $\mu=2$, ... – Scortchi - Reinstate Monica Jun 19 '19 at 14:13
  • 1
    ... $\Pr[b_\mathrm{U}(\vec{X}_2) < 2 < b_\mathrm{U}(\vec{X}_2)]=0.95$ is true, & if $\mu=7$, $\Pr[b_\mathrm{U}(\vec{X}_7) < 7 < b_\mathrm{U}(\vec{X}_7)]=0.95$ is true. Now substituting in realized values of $X_\mu$ gives e.g. $\Pr[1.2 < \mu < 3.4]=0.95$, i.e. if $\mu=2$, $\Pr[1.2 < 2 < 3.4]=0.95$ & if $\mu=7$, $\Pr[1.2 < 2 < 3.4]=0.95$ - which is nonsense. – Scortchi - Reinstate Monica Jun 19 '19 at 14:13
  • Thank you for the explanation. It still seems to me that this is mostly a semantic problem. I mean if one interpretation of a statement is nonsense, and the other isn't, then it is likely that the second one was the intended interpretation. – PoorYorick Jun 21 '19 at 07:23
2

It can be hard to mathematically characterize uncertainty, but I know it when I see it; it usually has wide 95% confidence intervals.

N Brouwer
  • 1,786
  • 2
  • 18
  • 32