0

Let's assume I have to point estimates with 95%-CI. The source can be from a simple computation of two samples or from a complex regression analysis. For example: odds ratio

  • (1) 2.0 (95%-CI: 1.4-2.9)
  • (2) 2.8 (95%-CI: 2.5-3.3)

(A) From one statistical point of view (hypothesis-testing) one can state that the there is no difference between these two estimates since the 95%-CI of estimate (1) does overlap the estimate of estimate (2).

(B) From a more relaxed view of statistics which sees statistics more as a method which delivers estimates and measurements of accurateness (95%-CI) one can state that estimate (2) "may be" higher than estimate (1) or (2) is "possible" higher than (1) since the confidence interval of (2) does not overlap the estimate of (1).

I found both version correct and in context of random control trials I would tend to (A) and in context of epidemiological studies to (B).

Concretely, if I would state in a epidemiological study:

(2) is possible higher than (1)

is this completely wrong?

giordano
  • 829
  • 8
  • 19
  • Confidence intervals are about parameters, not about estimates. So it might help to think about what parameter your desired hypothesis can be formulated. – Michael M May 05 '21 at 16:57
  • 1
    There are two problems with (A). 1. It is possible to have overlapping confidence intervals for values that are significantly different from one another. 2. A result that does not find a significant difference between two values does not conclude that there is no difference. It only tells you that you didn't have enough evidence to conclude that they were different. – CFD May 05 '21 at 17:09
  • @Michael M: Should I state "I have two point estimates and corresponding 95%-CI-estimates of two parameters a and b"? I abbreviated it to: "Two estimate with 95%-CI". – giordano May 05 '21 at 17:13
  • @CFD: ... and if I don't have enough evidence there is possible a difference? Can I state this if the difference of OR 2.0 and 2.8 is important but not enough evident? – giordano May 05 '21 at 17:16
  • 2
    In the context of a t-test -- which differs in no essential respects from this one -- this issue was thoroughly investigated at https://stats.stackexchange.com/questions/18215. The bottom line is that the overlap of confidence intervals *can* be used to perform the kind of heuristic test you propose, but the p-value needs (substantial) adjustment. – whuber May 05 '21 at 18:33
  • @whuber Thanks for this very useful link. In Cummings and Finch's paper there is rule of eye 2 (Make a substantive interpretation of the means): "The first interpretive focus should be the means, or combinations or pattern of means, and these should be assessed against any theoretical prediction. Use knowledgeable judgement in the research situation and consider the extent to which an effect is (a) important or interesting and (b) large. Distinguish practical or clinical significance from statistical significance." I could state: there is a possible important difference between (1) and (2). – giordano May 06 '21 at 07:09

1 Answers1

1

A useful approach would be to compute an effect size, such as Cohen's d.

Then you could report the size and direction of an effect size and its 95% confidence interval.

Cohen's D = (mean1 - mean2) / (pooled_standard_deviation)

pooled_standard_deviation = sqrt((standard_deviation_1^2 + standard_deviation_2^2) / 2)

Some statistical programs will give you a 95% CI on the effect size too, so you would have a single CI to interpret rather than two.

sharoz
  • 193
  • 11
  • Or as alternative instead of Cohen's D: difference of the two means and a pooled 95%-CI of the difference. But: how would you explain that to someone who had 6 years of school education? – giordano May 06 '21 at 06:53
  • Then maybe drop the numbers, and go with a distribution of simple effect size. Show something like a histogram, and explain that each bar shows how likely and by how much [whatever you're comparing]. Histograms are often pretty understandable even to people without much stats training. – sharoz May 06 '21 at 17:49