22

Suppose I am a consultant and I want to explain to my client the usefulness of confidence interval. The client says to me that my intervals are too wide to be useful and he would prefer to use ones half as wide.

How should I respond?

Macro
  • 40,561
  • 8
  • 143
  • 148
student_diversified
  • 217
  • 2
  • 4
  • 15
    bill them to collect more data. – shabbychef Jun 24 '12 at 04:55
  • 2
    This reminds me of a [must-read- paper](http://www.jstor.org/stable/2528938) about types of clients in statistical consultancy. –  Jun 24 '12 at 08:47
  • @Procrastinator Would you mind posting a link to a PDF version of the paper? – assumednormal Jun 26 '12 at 00:55
  • 1
    @Max, it seems only to be obviously available through JSTOR; posting a PDF would be a violation of JSTOR's (quite reasonable) terms of service ... – Ben Bolker Jun 26 '12 at 09:18
  • @shabbychef - often this is a highly inefficient way to increase accuracy, especially if the sample is already large. for example, to halve the width of the ci of a normal mean ci you need to quadruple the sample size. better to spend some time improving the model before you go and increase you largest cost fourfold! – probabilityislogic Jul 05 '12 at 09:53

4 Answers4

16

It depends on what the client means by "useful". Your client's suggestion that you arbitrarily narrow the intervals seems to reflect a misunderstanding that, by narrowing the intervals you've somehow magically decreased the margin of error. Assuming the data set has already been collected and is fixed (if this isn't the case, @shabbychef's joke in the comments gives you your answer), any response to your client should emphasize and describe why there's no "free lunch" and that you are sacrificing something by narrowing the intervals.

Specifically, since the data set is fixed, the only way you can reduce the width of the confidence interval is by decreasing the confidence level. Therefore, you have the choice between a wider interval that you're more confident contains the true parameter value or a narrower interval that you're less confident about. That is, wider confidence intervals are more conservative. Of course, you can never just optimize either width or confidence level mindlessly, since you can vacuously generate a $100 \%$ confidence interval by letting it span the entire parameter space and can get an infinitely narrow confidence interval, although it will have $0 \%$ coverage.

Whether or not a less conservative interval is more useful clearly depends both on the context and how the width of the interval varies as a function of the confidence level, but I'm having trouble envisioning an application where using a much lower confidence level to obtain narrower intervals would be preferable. Also, it's worth pointing out that the $95 \%$ confidence interval has become so ubiquitous that it will be hard to justify why you're, for example, using a $60\%$ confidence interval.

Macro
  • 40,561
  • 8
  • 143
  • 148
  • 6
    You can't simply shrink the confidence interval without giving something up, but there is a little flexibility analogous to the difference between a one-tailed test and a two-tailed test. In addition, it is possible that a better model of the same data would produce different (and possibly smaller) confidence intervals. – Douglas Zare Jun 24 '12 at 11:56
  • This is a good point @DouglasZare - I suppose I was assuming the methodology was also fixed. If your client is keen enough to suggest a more efficient method for obtaining confidence intervals (for a fixed confidence level) then perhaps he/she should be consulting you! – Macro Jun 24 '12 at 14:16
  • 3
    I think you're selling the last part just a little bit short. The ubiquity of the **95** % CI is a cultural phenomenon. In some contexts other values are common, eg when plotting, error bars are often standard errors (ie 68% CI's), as I'm sure you're familiar. (+1, btw) – gung - Reinstate Monica Jun 24 '12 at 20:21
  • 2
    In spirit this reply is good but I think the second paragraph is too limiting. One *huge* benefit the statistician brings to this party is knowledge of alternative procedures that can better meet the client's needs. In many cases one can narrow the width of a CI by selecting a different CI procedure. This neither requires collecting more data (-1 to @shabbychef, I'm afraid) nor decreasing the confidence level. The really hard part is *interpreting* a CI where the procedure was selected *post hoc.* That's why we want to have this conversation before analyzing (or even collecting) the data! – whuber Jun 25 '12 at 13:17
  • @whuber, my thinking was in a similar vein to your comment **"The really hard part is interpreting a CI where the procedure was selected post hoc"** which is why, when I wrote this, I was assuming the methodology had been fixed as it, hypothetically, should be before you start analyzing the data. Once you're at the point where you're looking at a CI and thinking it's too wide, it _should_ be too late to make such a shift. – Macro Jun 25 '12 at 13:20
  • 7
    I'm reacting to this, Macro, because in practice it does not work well to be so inflexible. The risk is that the client will ignore your advice and turn for help from someone who does not know any better (but claims sufficient expertise in statistics). The OP's scenario is a familiar and common one: it's best seen as an opportunity to inform and educate the client as well as offer them alternatives (along with a frank discussion of their pros and cons). We need to say "yes, the CI can be made smaller, but here are some of the consequences of doing that" rather than "no, you're screwed." – whuber Jun 25 '12 at 13:26
  • 2
    That's a good point @whuber (+1) in situations where more efficient alternatives may be available - yet another reason to consult with a statistician **before** collecting/analyzing the data. – Macro Jun 25 '12 at 13:40
  • @gung: I agree that what confidence level is used needs to be thought about, and we can't assume that 95% is sacred. I've seen 68% and even 50% (easily interpreted). Perhaps exploring the use is also important. For example, if I'm trying to budget for electricity expenses for my organization for next year and hear a forecast of \$200,000 +/- \$150,000, my immediate feeling is that you haven't really told me anything useful. That's when talking about risk and decision-making might give a more meaningful answer. – Wayne Jun 26 '12 at 00:53
2

I would suggest it entirely depends on what your client wants to use the confidence intervals for.

  1. Some sort of report/publication/etc. where 95% CI's are normally reported. I might very well tell him "That's not statistically justified" and leave it there, depending on whether or not the client tends to defer to your expertise. If they don't, you have to make a judgement about your own professional comfort with what they want.
  2. Some sort of internal document - I'd make it clear you disagree, and make it clear what type of confidence interval the reader is now looking at, since it's not 95%.
  3. As a measure of estimate uncertainty, say to determine how much sensitivity analysis one might have to do? I'd give them a figure showing the full distribution with both the 95% CI and something like a 68% CI marked and let them have at it.

I would be quite proud of myself if I managed to keep "So run a bigger study" from being the first thing out of my mouth.

Macro
  • 40,561
  • 8
  • 143
  • 148
Fomite
  • 21,264
  • 10
  • 78
  • 137
  • 1
    +1. I think the comments you made under (2) would probably be relevant in the situation described by (1) as well. – Macro Jun 25 '12 at 12:23
0

Use the Standard Deviation, as most people do. 95% CI can be scary when people are used to the 68%CI.

Geoph Twombly
  • 1
  • 4
    It sounds to me that in this case we are merely interested in showing the accuracy of, say, the sample mean, not the variability of individual values. Why would you recommend *standard deviation*, specifically? – chl Jun 24 '12 at 08:51
  • Fisher originally suggested 95% CIs as an approximation to 2 standard deviations. – Patrick Caldon Jun 25 '12 at 06:27
  • 1
    @Patrick, it sounds like either you missed chl's point (as well as mis-represented Fisher, who made no such mistake) or else you wrote "standard deviation" where you intended "standard *error*". Most CIs are based on standard errors, of course, not standard deviations. 2 SDs neither approximates a CI nor *vice versa.* – whuber Jun 25 '12 at 13:21
  • Of course, the standard error is just the standard deviation of the mean, so it's just terminology. That is, saying that CIs aren't based on standard deviations isn't really true. They're not based on the standard deviation of the sample, but instead on the standard deviation of the mean. – Aaron left Stack Overflow Jun 25 '12 at 20:00
  • For an illuminating counterexample to your assertion @Aaron, please see http://stats.stackexchange.com/a/1836. – whuber Jun 25 '12 at 20:10
  • Interesting! I suppose I should have been more precise and referred to Wald-type CIs, or something like that. – Aaron left Stack Overflow Jun 25 '12 at 20:21
  • 2
    Not all estimates are means. There are standard errors for estimates other than means and it is the standard error of the estimate that is used to generate confidence intervals for a parameter based on the variability of an estimate as whuber suggests. – Michael R. Chernick Jun 25 '12 at 22:56
  • @whuber - my mistake, sloppy reading on my part – Patrick Caldon Jun 26 '12 at 02:40
0

You provide a confidence interval at a certain standard level such as 90% or 95%. The client can judge whether or not the interval is too wide to be useful. But of course that does not mean that you can shorten it to make it useful. You can suggest that increasing the sample size will decrease the width of an interval at a given confidence level as it decreases roughly by a factor of the square root of the sample size.

Michael R. Chernick
  • 39,640
  • 28
  • 74
  • 143