Difference between fitting one predictor and many predictors in GLM

Question

If I have this data,

y=c(0,1,0,1,1,1)
x1=c(0,1,2,2,1,0)
x2=c(1,2,2,2,0,0)
x3=c(1,2,2,2,1,0)
x4=c(0,0,0,0,1,0)
model1<-glm(y~x1,family=binomial)
model12-glm(y~x1+x2+x3+x4,family=binomial)

If we fit $y$ with just $x_1$ we will have a coefficient for $x_1$, namely $\hat{\beta_1}$, and if we fit $x_1$ and all the $x$s together (jointly) we will have four coefficients, $\hat{\beta_1}$, $\hat{\beta_2}$,$\hat{\beta_3}$, $\hat{\beta_4}$.

My question is what is the difference between the value of coefficient of $x_1$ in model1 and $x_1$ in model2? (Difference between $\hat{\beta_1}$ in model 1 and $\hat{\beta_1}$ in model 2)

Answer 1

In logistic regression, the interpretation of coefficient $\beta_i$ is the log odds ratio for a 1 unit increase in dependent variable $X_i$ holding all other dependent variables in the model constant. When you have a different set of dependent variables in the model, the interpretation of the coefficients change. In particular, if you add dependent variables that are both correlated to the dependent variables already in the model and correlated with the response variable you will see changes.

Consider the situation where $Y$ is correlated with both $X_1$ and $X_2$ - which are both correlated with each other. As an example, the temperature ($X_2$) is correlated with the murder rate $(Y)$ and the temperature is also correlated with ice cream sales $(X_1)$. As a consequence, ice cream sales is also correlated with the murder rate. If only $X_1$ is in the model, some of the variation in $Y$ that is due to $X_2$ will be captured by $X_1$. However, when we add $X_2$ to the model none of the variation in $Y$ that is due to $X_2$ will be captured by the $X_1$ coefficient since we're controlling for $X_2$ in this new model!

Answer 2

As a very simplistic illustration, let's consider OLS linear regression with 0 and 1 independent variables, both with offsets (i.e. $\beta_0$). Note I choose these dimensions as 2D graphs are easier to view / understand than higher dimensional ones.

So we can generate two different models namely:

1. $\hat{y}=\beta_0$
2. $\hat{y}=\beta_0 + \beta_1x$

Now let's look at a plot of what betas would minimize the square errors for each model. Note that model 1 is shown in red, model 2 in blue:

You can draw a direct analogy to your question of why $\beta_1$ changes in logistic regression when adding more independent variables, and why $\beta_0$ changes in this toy example when we add a new independent variable (i.e. $x$ in equation 2). So in the case of both lines, $\beta_0$ is just where they cut the vertical axis. It is clear that adding an extra independent variable has considerably changed $\beta_0$. This is because your model now has an extra dimension on which it can try to minimize the errors. (Note that to be strictly fair, please think of the red line as just a single point on the vertical axis and when regressing for it, the black dots should also just be points on the y axis as this would be a 1D case, but that doesn't change anything).

So adding the extra independent variable allows your model an extra parameter it can tune (i.e. $\beta_1$) which "takes the pressure off" of $\beta_0$ as it no longer has to model the variation actually due to another dimension. When that dimension is not there, the model still does its best (i.e. minimizes square errors) but in order to do that $\beta_0$ needs to model the variation due to both dimensions alone.

Your question is in higher dimensions and uses logistic regression, but conceptually the reason that the $\beta$ changes is the same.