Coefficient changes sign when adding a variable in logistic regression

Question

In my logistic regression the sign of coefficients of a variable (location distance of an amenity) changes based on other variables (with time -ve, with travel distance +ve) in the model. When the location distance is the only variable in the model, it has +ve sign.

• Should the variable need to maintain the +ve sign no matter what other variables are added in the model?
• Does changing sign signify a multicollinearity issue? Some IVs are gaining significance while in a bivariate model, they didn't show significance and vice versa.
• Is it okay to add variables that don't have much significance (ex: travel distance has a significance of 0.33 individually, but 0.05 when added with other variables) but becomes significant in the model? thanks.

Answer 1

In addition to the links to Simpson's paradox in the comments, here is another way to think about it.

Imagine a dataset that is collected by counting the numbers and types of coins that various people have with them (I will use US Currency for the example, but it could be translated to other currencies as well).

Now we create 3 variables, the y variable is an indicator for whether the change totals to more than 1 dollar (\$1.00), x1 is the total number of coins and x2 is the total number of pennies (\$0.01) and nickels (\$0.05) (this will be a subset of x1). Now if regressed individually we would expect that x1 and x2 would have positive coefficients, the more coins, the more likely the total is over \$1. But if put into a regression model together then it makes sense for the coefficient on x2 to become negative, remember the definition of the individual coefficient is the change in y (or in the logistic case the change in the log odds of y) for a 1 unit change in x while holding the other variables constant. So if we have the same number of total coins (x1) but increase the number of small value coins (x2) then we have fewer of the large value coins and so a smaller chance of totaling over \$1.

Answer 2

Predictors do change their signs in the presence of others in a model. I think you are seeing a special case of "suppression". Let me explain using correlations (this principle should be applicable to logistic regression). Say you are trying to predict the extent of fire damage done to a house ($Y$) from the severity of the fire ($X_1$) and the number of fire fighters sent to put out the fire ($X_2$). Assume $r_{YX_1}=0.65, \: r_{YX_2}=0.25, \: r_{X_1X_2}=0.70$. Then, if you compute semi-partial correlations,

$r_{Y(X_1X_2)} = \displaystyle\frac{0.65-0.25*0.70}{\sqrt{1-0.70^2}} = 0.67, \: r_{Y(X_2X_1)} = \displaystyle\frac{0.25-0.65*0.70}{\sqrt{1-0.70^2}} = -0.29$

This is a case of suppression (albeit very slight) because $X_2$ suppressed the variance unaccounted for by $X_1$, resulting in $r_{Y(X_1X_2)} > r_{YX_1}$. Also, $X_2$'s semi-partial correlation ($r_{Y(X_2X_1)}$) switched its sign because its positive correlation with Y was mainly through its large positive correlation with $X_1$. Conceptually this make sense: if fire severity is held constant, sending more firefighters should result in less damage to a house (Messick & Van de Geer, 1981).

In your case, you need to think whether it makes sense that, while holding the time variable constant, location distance of an amenity be negatively related to the dependent variable. I also suggest some great posts on this issue in Cross Validated

Answering your other questions, I do not believe your data are suffering from multicollinearity; otherwise, all predictors should show inflated standard errors and lower p-values. Finally, of course you can add the travel-distance variable to the model since it seems its true relationship was masked by irrelevant variance (which was 'suppressed' by other predictors).

It is really up to the original questions you were trying to answer by designing your study.

Reference

Messick, D.M. & Van de Geer, J.P. "A reversal paradox." Psychological Bulletin 90.3 (1981): 582.

Answer 3

In my logistic regression the sign of coefficients of a variable (location distance of an amenity) changes based on other variables (with time -ve, with travel distance +ve) in the model. When the location distance is the only variable in the model, it has +ve sign.

This isn't surprising. It happens in ordinary regression as well. See the example in the image here

Should the variable need to maintain the +ve sign no matter what other variables are added in the model?

I don't see why this would be expected to be the case.

Does changing sign signify a multicollinearity issue?

Not necessarily multicollinearity; it can occur with quite ordinary non-orthogonality.

Some IVs are gaining significance while in a bivariate model, they didn't show significance and vice versa.

Sure, also common.

Is it okay to add variables that don't have much significance (ex: travel distance has a significance of 0.33 individually, but 0.05 when added with other variables) but becomes significant in the model?

Sure. It's also okay to add variables that aren't significant in either case (though if you throw in a large number of them it can cause problems. However, it sounds like you're doing variable selection; be very cautious about interpreting p-values/test statistics when you do that.

Answer 4

I think this may be a case of ceteris paribus confusion. When travel distance is the only variable, the effect on the outcome is positive. If the outcome is a purchase, this might be explained by the fact that when an agent lives far away, a trip to the store is more expensive, so he is more likely to stock up if he's already there. People who live far away fill their carts all the way, but make fewer trips, compared to people who live closer. I would bet dollars to donuts this is also what you would find if you used only travel time in the model as your measure of cost.

When you have both travel distance and travel time in the model, the sign of the distance coefficient gives you sign of the effect holding travel time fixed. When distance gets longer, but the travel time stays constant, the effect becomes negative. How might distance get longer, but travel time remain the same? If the speed of travel on the road became faster, perhaps because it was a highway with a higher speed limit. The comparison you are now making when both variables are in the model is between two identical people who both live $X$ minutes from a store, but one lives further away and takes a highway to get there. That agent is less likely to make a purchase, perhaps because traveling on the highway is easier than taking the local roads on gas usage, or perhaps this is the road he uses to commute to work and he passes the store on the way home (a kind of omitted variable in your model).

To sum up, when the regressors are different, the coefficients corresponds to different thought experiment comparisons and the interpretation changes accordingly. The changing signs do not necessarily indicate multicollinearity. Variable selection should be driven by theory, careful thought, and your ultimate goals.

Answer 5

Nothing you said indicates to me that there is a problem with your models: they are all good answers to different questions. It is up to you to decide which question you want to answer, and thus which model you want to report.