12

If someone makes a statement like below:

"Overall, nonsmokers exposed to environmental smoke had a relative risk of coronary heart disease of 1.25 (95 percent confidence interval, 1.17 to 1.32) as compared with nonsmokers not exposed to smoke."

What is the relative risk for the population as a whole? How many things are connected with coronary heart disease? Of the vast number of things that can be tested, very few actually are connected to coronary heart disease, so the chance that any particular thing chosen at random is connected is vanishingly small. Thus we can say that the relative risk for the population is 1. But the quoted interval does not contain the value 1. So either there actually is a connection between the two things, the probability of which is vanishingly small, or this is one of the 5% of intervals that do not contain the parameter. As the latter is far more likely than the former it is what we should assume. Therefore, the appropriate conclusion is that the data set was almost certainly atypical of the population, and thus no connection can be implied.

Of course, if there is some basis for assuming that more than 5% of things are linked to coronary heart disease then there might be some evidence in the statistic to support the suggestion that environmental smoke is one of them. Common sense suggests that this is unlikely.

What is the error in their reasoning (as all health organizations agree that there is significant literature regarding the damaging effects of second-hand smoking)? Is it because of their premise that "Of the vast number of things that can be tested, very few actually are connected to coronary heart disease"? This sentence may be true for any randomly chosen factor (ie. how many dogs a person owns with the risk of coronary artery disease) but the a priori probability is much higher for second hand smoking and coronary heart disease than just 'any random factor'.

Is this the correct reasoning? Or is there something else?

Community
  • 1
BYS2
  • 1,365
  • 2
  • 12
  • 19
  • The quoted text seems like... well, like a quote. Where is it from? :) – MånsT Jul 11 '12 at 08:34
  • haha yes it is a quote, from wikipedia... someone added this to the article for "confidence interval". I'm trying to get it removed because that is clearly incorrect, but the guy refuses so I need a mathematically sound reason instead of just "this is clearly wrong".. although I have some ideas, I wanted to know if someone could explain exactly what errors are being made here. Because if this were correct then many studies can be refuted on similar grounds – BYS2 Jul 11 '12 at 09:24
  • 3
    If it drags on a bit, I'll move over and try to help out. His argument is clearly fallacious and points strongly at him having an agenda. – Erik Jul 11 '12 at 10:02
  • 3
    As a physicist who uses a lot of statistics but isn't a statitician I find that paragraph really unhelpful, never mind the fact it sounds plain wrong. I have always thought, possibly incorrectly, that a 95% c.l. meant that if the null hypothesis were true then 1 time in 20 if I repeated my experiment would I get a result significant at the 95% level (a good reason in my opinion to not use less than 99.9 but thats another discussion). That post seems to be more a point about correlated factors and really doesn't help non-experts (or anyone) at all. – Bowler Jul 11 '12 at 13:49
  • @Erik. The user has a pretty dodgy history of sock puppetry (had a few accounts and uses IP edits) and has gotten blocked before... not sure what his deal is. But does seem like a troublemaker – BYS2 Jul 12 '12 at 05:18
  • Wow all the answers have been great, now sure which one to choose as the 'correct' one :S. Might just do a random pick – BYS2 Jul 12 '12 at 05:21

5 Answers5

10

There are many things wrong here. As @Néstor explains, he implicitly assumes prior probabilities on $H_0$ (no link) and $H_1$ (link).

He places a very high weight (very close to 1) on $H_0$ and a very small weight on $H_1$. This is the first dubious thing he does, since there is a mechanistic link between smoke and heart disease (consider active smokers), the question really is if the exposure is enough. This does not even consider the previous studies done. So it's really not one of a "vast number of things" to be tested as wearing red socks for example would be. This means he already starts with a highly biased and not really justifiable prior.

He then updates his prior by stating that the probability of getting an 95%-confidence interval of not containing the true value has a probability of 5%. While this is true, this is not the chance of getting that particular interval under the assumption of the null hypothesis. Note that he would have treated a confidence interval of [1.17, 1.32] exactly the same as a confidence interval of [100, 200] which is clearly problematic.

This is really important to the Bayesian approach: while you have a total probability of 5% of not getting an interval containing the 1 under the assumption that 1 is zero, the probability density of getting that particular interval is different (and smaller).

The third mistake is that he never specified his prior nor states how likely $H_0$ has to be versus $H_1$ for him to get that result. It's just "vanishingly small".

The fourth mistake is to say that the appropriate action to take would be to dismiss the data. Note that his result does not even depend on the data, his argument implies exactly the same action would have been done for any data at all. If you find an interesting link but suspect it might just be a fluke, the proper scientific thing to do is to try to replicate your result!

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
Erik
  • 6,909
  • 20
  • 48
  • Thanks for expanding on Nestor's answer! A quick question though, you stated that "... this is not the chance of getting that particular interval under the assumption of the null hypothesis." If we wanted to find the chance of getting a particular interval under the assumption of the null hypothesis, we would need to use bayesian inference and a credible interval correct? Frequentist confidence intervals only tells you "the chance that the interval will include the true value". Thanks again – BYS2 Jul 12 '12 at 05:48
  • The frequentist confidence interval of 95% is constructed so that at least 95% of the times the interval contructed contains the true value. So far so good. This being said you can also compute the probability (or the value of the density) of getting a particular confidence interval if the null hypothesis is true. The exact location contains more "information" than just whether it included the null hypothesis. Throwing away that information is bad when using Bayesian inference as it relevant to the probability of the null being true. – Erik Jul 12 '12 at 06:42
  • A toy example would be this: Bayesian inference, you want to make inference toward the form of a distribution. Prior allows two possibilites : H1: Distribution is standard normal. H2: Distribution normal, mean = sd = 1. A sample of the values of the distributions allows you to update your prior. When you are given just the signs of your values you can also update your prior, but the update will be less informative since you threw away relevant information. – Erik Jul 12 '12 at 06:51
7

This is a quite interesting philosophical issue related to hypothesis testing (and thus in the frequentist setting also confidence intervals, as I explain here).

There are, of course, a lot of hypotheses that could be investigated - passive smoking causes coronary heart disease, drinking alcohol causes chd, owning dogs causes chd, being a Capricorn causes chd...

If we choose one of all of these hypotheses at random, the probability of us choosing a hypothesis that happens to be true is virtually zero. This seems to be the argument in the quoted text - that it is very unlikely that we happened to test a true hypothesis.

But the hypothesis was not chosen at random. It was motivated by previous epidemiological and medical knowledge about coronary heart disease. There are theoretical mechanisms that explain how smoking could cause coronary heart disease, so it does not seem far-fetched to think that those would work for passive smoking as well.

The criticism in the quote may be valid for exploratory studies where a data set is mined for hypotheses. That is the reason that we don't accept such "discoveries" as facts - instead we require that the results can be replicated in new studies. Either way, the paper cited in the quote is a meta study and is therefore not affected by this problem.

We have seen empirically over the last centuries that testing hypotheses motivated by theory by comparing the predicted results to the observed results works. The fact that we believe in this procedure is the reason that we have made so much progress in medicine, engineering and science. It is the reason that I can write this on my computer and that you can read it on yours. To argue that this procedure is wrong is to argue that the scientific method is fundamentally flawed - and we have plenty of evidence that says otherwise.

I doubt that there is anything that a person who isn't willing to accept this kind of evidence actually will accept...

Community
  • 1
MånsT
  • 10,213
  • 1
  • 46
  • 65
  • I really didn't get your paragraph before the last; are you refering to "significance testing" (e.g., calculating the probability of data at least more extreme) or really to "hypothesis testing" (the bayesian setting)? Who said that any of them doesn't work if you ask the right question? – Néstor Jul 11 '12 at 17:37
  • @Néstor: I should perhaps have written that differently. I wasn't really making a statement about _statistical_ hypothesis testing, but rather making an observation about the fact that comparing model predictions with real-world data (i.e. "testing" if the hypothesis is correct) seems to be a very efficient way of doing science. At the heart of this criticism against CI's is, I believe, an unwillingness to accept this method. The kind of arguments given in the quote would apply to any statistical method - with zero prior probabilities for all null hypotheses, we'd never believe in anything. – MånsT Jul 12 '12 at 06:49
6

I really don't get why the author says that the probability of relative risk of coronary heart disease being 1 could be vanishingly small basing his analysis solely on a Confidence Interval; this is plain wrong. To me, it looks like he's using a frequentist setting, but he is reasoning bayesianly (which is pretty common).

The only thing linked to a C.I. are classical significance tests but, as we all know, if $H_0:$ { There is no link between second hand smoking and coronary heart disease }, they give you $p(D_e|H_0)$ (where $D_e$ denotes "data at least as extreme as what we observed"), not $p(H_0|D)$ (where $D$ is the data), which is what he claims, and what is linked exactly to what you point out; you have to incorporate prior knowledge about that particular link! This comes from the fact that: $$p(H_0|D)\propto p(D|H_0)p(H_0),$$ by Bayes Theorem, where $p(H_0)$ is the prior probability on $H_0$.

Néstor
  • 3,717
  • 26
  • 37
  • Wouldn't H0 be: there is no link between passive smoking and CHD? Since the null hypothesis is usually the hypothesis that there is no effect. Apart from that though, thanks for this answer! – BYS2 Jul 12 '12 at 05:30
  • Yeah, you are right! I didn't noticed it until you pointed out :-). I'll edit my answer. – Néstor Jul 12 '12 at 05:41
3

While there is something to this Bayesian line of reasoning (deconstructed very thoroughly by Erik!), and indeed this line of thought would explain why many medical findings cannot be reproduced, this particular argument applies that thinking like a sledgehammer.

The author presupposes two things without providing evidence: that exposure to smoke was chosen at random, and that almost nothing in the world causes heart disease. Under these lax standards of reasoning, the author could reject ANY conclusion that something causes heart disease. All you would need to do is assert:

  1. That the hypothesis was chosen at random, and
  2. That heart disease has very close to zero causes.

Both of these assertions are debatable (and, based on my general knowledge, very likely false). But, with these assumptions in place, even observing that 100% of people exposed to secondhand smoke dropped dead of a heart attack within a year, you could assert that the connection is merely a coincidental correlation with the hidden, singular, "true" cause.

Jonathan
  • 1,283
  • 8
  • 15
  • Alright thanks for your thoughts! Yes, the author definitely assumed that the hypothesis was 'chosen at random' which is not correct. – BYS2 Jul 12 '12 at 05:38
-1

I don't see anything obviously wrong with the paragraph in quotations, but I haven't seen the data and cannot check is numbers. However, the two paragraphs that follow it are very unclear.

Suppose he had said, "Overall, nonsmokers who were morbidly obese had a relative risk of coronary heart disease of 1.25 (95 percent confidence interval, 1.17 to 1.32) as compared with nonsmokers who had normal body weight." Would anyone have reason to doubt him?

Emil Friedman
  • 711
  • 4
  • 6
  • Well the first paragraph in quotations was just the author quoting the conclusions of a epidemiological study so there is nothing wrong with it.. The next few paragraphs are him trying to discredit the study which is where he makes some dubious statements. – BYS2 Jul 18 '12 at 07:28