1

Possible Duplicate:
Meaning of 2.04 standard errors? Significantly different means when confidence intervals widely overlap?

Based on 3 different tests, I estimated the following return period values* and their 95% confidence intervals from the test measurements. As an example, imagine that we are testing different engine oils and measuring the engine temperature, and we want to know which oil results in the lowest engine temperature. The confidence intervals from the first 2 tests contain the return period value from the other tests. Does that mean that we can't actually distinguish the 3 return period values, i.e. the difference between the values is not statistically significant? If that interpretation is incorrect, what is the right interpretation of overlapping confidence intervals?

Value (Lower 95% CI, Upper 95% CI)
3.36 (3.09, 4.61)
3.50 (3.02, 7.33)
4.35 (4.06, 5.50)

*The return period value is just the value associated with a certain probability level. An example is the wind speed associated with a 100-year storm.

Community
  • 1
KAE
  • 423
  • 2
  • 15
  • Are the three different values just categorically different? Is there a reason that they should progress in value? What have you looked at to solve this? – John Jan 21 '13 at 17:08
  • In each test, some parameters were varied (an example might be trying different types of oil in car engines and looking at the engine temperature). Those parameters could cause a real difference in the return period value. Is that what you are asking? Or if not, please explain a little more. – KAE Jan 21 '13 at 17:20
  • This appears to be almost an exact duplicate of the second question in http://stats.stackexchange.com/questions/31657/meaning-of-2-04-standard-errors-significantly-different-means-when-confidence-i. If it is not a duplicate and your question is not answered in that thread, then please tell us how your situation is different. Also closely related: [comparing confidence intervals with two samples](http://stats.stackexchange.com/questions/18215/relation-between-confidence-interval-and-testing-statistical-hypothesis-for-t-te). – whuber Jan 21 '13 at 17:33
  • Yes, the content in my question overlaps those two questions. I am sorry to have wasted everyone's time, although the answers here were very useful for my understanding. Nonetheless, you are free to put it up for a vote for deletion as a repeated question. – KAE Jan 21 '13 at 18:17
  • 3
    There was no waste of time. We benefit by having a good question expressed in various ways, because that can enhance future search successes. When we close a question as a duplicate, it usually remains on our site--it is rarely actually *deleted*--and thereby becomes part of a Web of links among related questions. Finally, identifying the duplicates is one way we can get you a good answer really quickly. I am glad that it helped in your case. – whuber Jan 21 '13 at 18:28

1 Answers1

-1

The three values are different. There are then two follow-up questions:

1) Is that difference important? 2) Is that difference statistically significant?

these two are related, but not identical. The confidence intervals help to answer the second question (although there are tests which give p-values, if that's what you really want). They do not help with the first question.

The first question asks if the differences are ones that you would care about. The second question asks if these are differences that would be likely to arise if there were no difference in the population from which this sample was drawn.

You ask are they "really different".... well, what do you mean? You say you want to find the one with the lowest return: That is easy: The first one is lowest.

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
  • Thanks, very helpful. I would like to know #2, whether the difference between these 3 values is statistically significant, given the fact that the CIs for the first two tests overlap. I will edit the question to clarify. – KAE Jan 21 '13 at 17:22
  • The key question is not whether the CIs overlap, but what the p-value is. These are related but I don't have time right now to go into it. – Peter Flom Jan 21 '13 at 17:26
  • 1
    Your first and last lines may be perpetuating the very confusion I think you're trying to clarify, Peter. I interpret the question as asking whether or not we should conclude three *underlying parameters* are different, whereas your answer refers to rather obvious differences among three *sample statistics* intended to estimate those parameters. – whuber Jan 21 '13 at 17:37
  • I am not sure what the question intended; but the sample statistics are the best estimate of the underlying parameter, regardless of confidence interval - it's just that if the CI is very wide, the estimate isn't very good. So, if we had to pick one of these as having the lowest underlying parameter value, we would have to pick the first one. – Peter Flom Jan 21 '13 at 19:25
  • 1
    -1. I agree with @whuber here. The fact that the three sample statistics are different" is not very useful to point out - assuming these data are continuous, the estimates will be different with probability $1$. Also, I don't understand *"The key question is not whether the CIs overlap, but what the p-value is"*-these confidence intervals are based on inverting hypothesis tests; two CIs overlapping is equivalent to not rejecting the null comparing the two groups for equivalent means. Finally, I'm not sure what is the motivation for answering when you are "not sure what the question intended". – Macro Jan 23 '13 at 16:29