1

I've made a bar plot and I calculated confidence intervals by hand, 

(+/- 1.96*std_error)+predicted_probability

and I want to be sure that it's interpretable. That is, do they have the same interpretation that they would if the point estimate was a mean or a logit? I understand that CIs for odds ratios and probabilities are inherently non-symmetric, but does this affect their interpretation?

For example, if I had two means and 83% of the total length of their CIs were non-overlapping, then the means are significantly different at .05. Would the same be true of CIs for predicted probabilities?

PanPsych
  • 440
  • 4
  • 13
  • 1
    "if I had two means and 83% of the total length of their CIs were non-overlapping, then the means are significantly different at .05" - not true. I think you mean that if you had estimates of two different means with the same standard error, non-overlap of the usual normal-theory CIs with 83% coverage around each would imply a p-value of less than 0.05 for the test for difference of the means. – Scortchi - Reinstate Monica Jun 12 '16 at 10:39
  • You say you know the CI should be asymmetric but the formula you are using gives rise to a symmetric interval. It would be better to use the Wilson method to get the CI. – mdewey Jun 12 '16 at 12:58
  • @mdewey, I don't follow. When using this formula with log odds and probabilities the results are asymmetric, which is not surprising. My question is whether this means reporting these CIs is problematic. Can you explain why the Wilson method is preferable? – PanPsych Jun 12 '16 at 14:54
  • If you use your formula directly with probabilities (which was your final question) then the interval will be symmetric. If you have transformed the probabilities in some way you did not explain and then back-transformed the limits of the interval then fine. – mdewey Jun 12 '16 at 14:56
  • @Scortchi, I probably didn't explain that well but here it was explained to me: http://stats.stackexchange.com/questions/203744/possible-to-have-overlapping-95-cis-for-two-means-but-95-ci-for-difference-betwe/ But my main question is whether what is true for means would hold for predicted probabilities (from a logistic regression)? – PanPsych Jun 12 '16 at 14:59
  • @mdewey, okay, that explains it. So then is this an appropriate approach (using the formula directly with probabilities)? – PanPsych Jun 12 '16 at 15:09
  • No it is not appropriate because the CI obtained by using your formula can give rise to impossible limits and will not have the stated coverage. – mdewey Jun 12 '16 at 15:28
  • @mdewey, thanks. I hadn't wrapped my head around it fully and when I went to calculate it I saw the problem all too clearly. Do you have any suggestions for how I can best address this (i.e., how I can generate CIs that are informative)? – PanPsych Jun 12 '16 at 15:52

1 Answers1

2

The issue of how to generate appropriate confidence intervals for a single proportion has been addressed repeatedly in the literature. A good example is

@ARTICLE{newcombe98a,
  author = {Newcombe, R G},
  year = 1998,
  title = {Two-sided confidence intervals for the single proportion:
          comparison of seven methods},
  journal = {Statistics in Medicine},
  volume = 17,
  pages = {857--872}}

the first of a series of three papers in which he also discusses confidence intervals for differences. He restricts himself to methods which do not involve iteration and compares among other the Wald method (which is what the OP suggests), Wilson's score based method and the Clopper-Pearson method. He concludes that the Wilson method has good properties.

As for the question of what they mean the answer is that they are confidence intervals and mean the same as any other confidence intervals: in the long run they will contain the true value with the specified coverage.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
mdewey
  • 16,541
  • 22
  • 30
  • 57
  • Thank you! I'd like to use Wilson's method. Any suggestions about how to do this in R using estimates generated from glmer? Or is it simple enough to do by hand? I'm actively looking into this but if you know, that would be great. I might have to open another question. – PanPsych Jun 12 '16 at 20:57