0

I am doing a regression analysis on the various factors which influence accident levels in my city. The 2 factors used in my regression model are covariates :

i) urbanization level of the city and,
ii) the percentage of persons buying cars within the past year in the city.

From my regression analysis, both showed significant positive association with my outcome variable.

However, when I presented my regression model, one of my colleagues mentioned that my analysis, i.e., the impact of the 2 covariates on the dependent variable could be flawed and does not reflect reality since I have left out an important covariate such as the number of alcohol outlets/pubs in the city. What statistical reasoning can I rebuttal his argument with? I am not quite sure what to make of his argument and if it is even valid. Suggestions/discussion are welcome. Cheers.

user121
  • 289
  • 2
  • 14

2 Answers2

4

@PeterFlom answer is very good and explains very well the general phenomenon known as Simpson's paradox. However, he forgot to mention one important thing, that is actually fundamental to our job: to be a confounding variable (one that, when omitted, invalids inference in linear models) a variable has to:

  • be associated with outcome (and this is the case);
  • be correlated with the studied variables.

In your case you could argue that drinking habits should not be dependent on urbanization nor with number of people buying cars (or maybe correlation exists, but is negative, in that case real effect of your study variable should be even stronger).

Also causal graphical models can be used to decide what variables to include in the model, so that their effect is evaluated without the influence of factors supposed to be caused by it, and not causing it.

Of course you can never guess the real, intricate relation among variables that cause every crash, and their aggregate measure. Linear models are always approximations, but with some solid reasoning, even littlest models can be shown useful.

carlo
  • 4,243
  • 1
  • 11
  • 26
  • would confounding variable that is correlated with other studied variables introduce multicollinearity and possibly lead to a flawed model? – user121 Oct 27 '19 at 14:05
  • If it is much correlated, that can happen, yes. Although moderate collinearity is very common and is not a problem. It may be worth pointing that collinearity inflates variance of $\hat \beta$, while omitted variables introduce bias, which is generally more frowned upon. Also, extreme collinearity can actually bring to wrong conclusions, but it is easier to deal with. For unknown variables you can only make assumptions like independence. – carlo Oct 27 '19 at 15:14
  • 1
    Lack of correlation of omitted predictors with included predictors does not prevent [omitted-variable bias in logistic regression](https://stats.stackexchange.com/q/113766/28500) or in other types of generalized linear models or Cox proportional hazards survival regression. You are correct for ordinary linear regression; I just add this comment for the benefit of others who might read your answer and be tempted to over-generalize. – EdM Oct 27 '19 at 16:18
  • 1
    This is true but not really important. Conditional odds ratios are not equal to unconditional ones, hence a logistic model omitting some variable will give different values for $\beta$ even if the omitted variable is independent on the others. But this is fine, as $\beta$ has in fact a different meaning among different models. – carlo Oct 27 '19 at 23:15
3

In general, yes, omitting a variable can lead to a totally wrong model and wrong conclusions. Three examples:

  1. If you are modeling the amount of damage done by a fire and you have a variable "number of firefighters sent" you will find a strong positive relationship. More firefighters, more damage. The omitted variable is size of fire.

  2. If you are modeling student grades and have a variable "did they hire a tutor" you will find that kids who hire tutors get worse grades than those who do not. You have omitted their grades before they hired a tutor (students who get great grades usually don't hire tutors).

  3. If you are modeling the vocabulary of 1st graders in a particular city, you will find a relationship with astrological sign. The variable you are omitting is "number of months in school".

Is your case like any of these? I don't know. The only way to be sure is to add "number of bars/pubs" to the model and show that the rest of the model stays more or less the same. You could try to make a substantive argument. His argument seems like a good one to me, just on my intuition. But you might try something like "the number of bars per capita doesn't really vary by city". As an aside, you might also want to look at the ages of car owners, as younger drivers have more accidents).

This is a general problem with observational studies - there could always be an important variable missing from the analysis.

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
  • 2
    If you also mention, too, that a good statistical and scientific mindset isn't to immediately rebut the colleague's argument, but as Peter Flom has said, think critically about the problem. In general, don't think of how to support your position or argument as you should focus more on what could be wrong with what you've done or tried to conclude. We can find ways to justify things, especially post-hoc, but failing to poke holes in our own arguments is poor practice. Long story short: in any project, don't look first at how well you've done, but rather at how poorly you might have done. (1/2) – LSC Oct 27 '19 at 09:51
  • 1
    (2/2) The failure to look meaningfully at issues in our work leads to the self-serving, pathetic "limitations" sections in each paper where the author just says "our sample size was small", "we try too hard and care too much", while omitting the obvious statistical or design issues. – LSC Oct 27 '19 at 09:53