16

I am aware that there are many, many threads on this (e.g. this excellent thread). I may have missed it but I can't seem to find one that actually explains how to accurately report a frequentist confidence using the actual numbers contained in the interval.

So say I have coefficient from a regression $\beta = 3.4$ with $CI = [0.5, 5.6]$

Bayesian Credible Interval

If the CI were a Bayesian Credible interval reporting this interval is quite straightforward:

"Given the data and the assumptions of the model, there is a 95% probability that the true value of $\beta$ lies between 0.5 and 5.6"

Frequentist Confidence Interval

Now as we know when the interval is a confidence interval reporting it properly is more tricky. Based on what I've read I would hazard

"If we ran many experiments 95% of the 95% intervals constructed would contain the true value of $\beta$"

What confuses me is that the actual numbers in the CI do not appear in this interpretation.

How do I work either the mean or CI into my reporting of the effect while staying faithful to the frequentist view of probability?

llewmills
  • 1,429
  • 12
  • 26
  • How do you figure that 95 isn’t a number? (I think I see it now, but that’s a clarification worth making in the post. As one of our newest mods like to say, not everyone reads the comments!) – Dave Feb 23 '21 at 01:06
  • Thanks Dave. Fixed it up – llewmills Feb 23 '21 at 04:07
  • The statement on the frequentist interval is trying to leverage the concept of coverage probability. One must know the relationship between coverage probability and confidence coefficients for their chosen interval generating procedure before accurately making this kind of statement. – Travis L Feb 23 '21 at 12:31
  • 1
    Also, the interval should contain the point estimate (if point estimate is 3.4, interval estimate should contain 3.4). – Travis L Feb 23 '21 at 14:35

4 Answers4

12

There are various ways you can reasonably word a confidence interval statement, but any variation on the statements below would be fine. (Since you did not specify to the contrary, I am assuming that this was a 95% confidence interval. If not then you should make the appropriate changes in the statement.) What is important is that you clearly give your confidence level and the interval of interest. You should also make sure that you properly distinguish between notation for estimates and notation for true parameter values.

  • Our estimate of the slope is $\hat{\beta} = 3.4$ ($\text{95%CI} = [0.5, 5.6]$).

  • With 95% confidence we infer that the true slope value is somewhere in the interval $0.5 \leqslant \beta \leqslant 5.6$.

  • With 95% confidence we find that $0.5 \leqslant \beta \leqslant 5.6$.

The concept of "confidence" has a clear and well-known meaning in classical statistics, so if you refer to having a certain level of confidence that a parameter falls within an interval then that will automatically be understood as reporting a confidence interval. There is no need for you to worry about staying faithful to a particular framework --- there is only one framework of probability theory used in practice$^\dagger$ and only one meaning of a confidence interval. Indeed, when other frameworks develop analogous ideas in (e.g., "credible intervals" in Bayesian inference) they make sure to use different terminology precisely to avoid confusion on this topic.

If you are just doing applied statistical work then there is no need to specify the exact statistical meaning of "confidence". It is subtle and confusing to most readers, and you can reasonably put the onus on them to read up on it if they are interested. Many people with an applied science background in any field will have done some introductory statistical courses, where they learned and then forgot what a confidence interval means. Most readers will just be satisfied with the fact that the statistical profession has given their imprimatur to the concept. For statistical experts, they will know the exact meaning of the concept, and be happy with your summary report just the same.

Worrying about the proper interpretation of "probability" is another order removed from this, and it is certainly not something you need to concern yourself with in reporting statistical analysis of data. Interested readers can dive down the rabbit-hole of philosophy and the foundations of probability theory if they really wish to do so.

$^\dagger$ There are occasional papers in academic literature that examine non-standard versions of "probability" (e.g., using complex numbers, etc.) but these are basically all bunk.

Ben
  • 91,027
  • 3
  • 150
  • 376
  • Only the first of your 3 formulations gave the 3.4 figure. Do you imagine that figure would be given in the previous sentence, or is there an argument it shouldn't be given at all? (As it is not in the exact middle of the interval it seems to be useful to specify it...) – Darren Cook Feb 25 '21 at 18:37
  • 1
    I would expect that you would give it in a previous sentence. – Ben Feb 26 '21 at 00:19
9

I agree with Ben that normally, you would just say, "Our estimate of the slope is $\hat{\beta}=3.4\ (95\%\ {\rm CI}=[0.5,5.6])$."

If you really need to explain to someone what a confidence interval is, I would probably not use your formulation, "If we ran many experiments 95% of the 95% intervals constructed would contain the true value of $β$." I do say that when I teach (although I supplement that with additional explanations and activities), but I never say it to clients. That just isn't a statement that works for many people.

To be honest, it's uncommon that I find I need to provide a definition for a confidence interval. If I do, what I say is some version of either:

If our null hypothesis had been any value within this interval, we would not have been able to rule it out.

or the inverse:

If our null hypothesis had been any value outside this interval, we could have ruled it out.

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
7

This post is such a goldmine if information. Its worth a read on its own.

Confidence interval interpretation is a practice in the Precision-Usefulness trade off: You can state what a confidence interval is precisely, and it will not be useful. As you relax the precision, it becomes more and more useful. However, this loss of precision is a cost, and the more useful you make your interpretation of a confidence interval the more people will disagree with you (due to the loss of precision).

One popular way of interpreting a confidence interval is as follows (as per that linked post):

The compatibility interval indicates a wide range of plausible true treatment effects.

Or put differently, "those parameters within the interval indicate plausible true values of the parameter". I'm not completely sold on this interpretation. Here "true" depends on the selected $\alpha$, and I don't find it compelling to take $\alpha = 0.05$ as corresponding to truth. With that in mind, perhaps a good revision would be

The compatibility interval indicates a range of plausible true treatment effects in accordance with our allowed false positive rate.

If its one thing I've learned doing and tweeting about statistics, its that people still disagree on the interpretation of the confidence interval. To avoid getting into conversations about this (again and again and again), I simply state that

The confidence interval summarizes regions of parameter space which are consistent with the data

I find this interpretation is a sufficiently good balance of usefulness and precision.

Demetri Pananos
  • 24,380
  • 1
  • 36
  • 94
  • Thanks Demetri Pananos, much appreciated. So are you saying that one should avoid stating the numerical upper and lower bounds of the confidence interval when verbally reporting the effects in question? – llewmills Feb 23 '21 at 05:18
  • 3
    No, I think that it is fine to say something like "We estimate the effect to be X (95% confidence interval (Y, Z))". If you *needed* to add an interpretation, then that is where the tricky part starts. – Demetri Pananos Feb 23 '21 at 05:20
  • 1
    Haha, yes I occurred to me while composing this that in practice one would never need to report a confidence interval verbally. As you say you would just report the point estimate and put an abbreviated CI in brackets. I was just wondering how you'd do it if, say, a weird statistical terrorist put a gun to your head and threatened to shoot you if you either refuse to do it or get it wrong. – llewmills Feb 23 '21 at 05:23
  • 2
    I'm very confident that -- gun to my head -- I would fail to give the correct interpretation of a confidence interval since, in my experience, someone always has something to say about everyone's interpretation. – Demetri Pananos Feb 23 '21 at 05:25
  • 2
    Ok, good to know. So if it happens to me I should play for time. – llewmills Feb 23 '21 at 05:40
  • +1, I *really* like the use of the word "compatible" here. I'm surprised I haven't seen it (or thought of it myself) before. – gung - Reinstate Monica Feb 23 '21 at 07:43
  • If you want to talk about plausibility etc or truly feel there is a precision/usefulness trade-off with confidence intervals, stop using them and start using Bayesian techniques! – innisfree Feb 24 '21 at 06:03
4

How about:

"A procedure which constructs an interval from an experiment, having the property that the interval constructed by the procedure contains the true value of $\beta$ in 95% of cases, was applied to this experiment, for which the estimated value of $\beta$ was $\widehat{\beta} = 3.4$. The interval constructed by the procedure was $[0.5, 5.6]$."

Flounderer
  • 9,575
  • 1
  • 32
  • 43