2

I was performing logistic regression on some data, and I realised that I need to remove or partial out the effects of another covariate.

If $x$ is the predictor of interest, the model is:

$$ y \sim \mathbf{logit}^{-1}(ax+b) $$

One option would be to include the covariate $z$ as another regressor:

$$ y \sim \mathbf{logit}^{-1}(ax+b + cz) $$

But in general $z$ might be correlated with $x$. So for my purposes, it is important to first regress out $z$. I don't want the parameter of interest $a$ to reflect any variance that could be explained by $z$.

If this were linear regression, I'd take the residuals from the regression $y\sim cz+b$, then regress these residuals against $x$: $(y-\hat{y})\sim x$ to obtain the parameter of interest. (By the way, is there a name for this process, of regressing out a variable that is not of interest?)

But for a logistic regression, I am not sure how to treat the residuals from the first regression. Say I do $\hat{y}=cz+b$. Can I attempt a logistic regression on the (non-binary) residuals $(y-\hat{y})$? Or perhaps perform a linear regression on some transformed function of the residuals? I am thinking of something like:

$$\log\bigg(\frac{\hat{y}-y}{y-\hat{y}}\bigg)$$

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
Sanjay Manohar
  • 906
  • 9
  • 16

1 Answers1

4

You don't do this. You shouldn't do what you are describing in a linear regression context either. All you need to do is include both variables in a multiple regression (multiple logistic regression) model. That will take care of this for you. Moreover, it won't matter if $x$ is correlated with $z$. If they are, then the standard errors will be larger (appropriately), but the estimated coefficients will be correct.

You may be interested in reading:

Community
  • 1
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
  • Dear Gung, thank you very much for the reading list. I can see now that the topic is complex. Two of the answers suggested the residual method, but further reading demonstrates that this is not always good when you don't know the underlying causal model. Simpson's paradox was particularly relevant. In my case, I know *for certain* that the confounder has a simple and strong causal effect upon the measurable. Therefore I considered **Gram-Schmidt orthogonalisation**, putting the confounder $z$ first. Now the $x$ coefficient reflects only its independent contribution. Does this sound reasonable? – Sanjay Manohar Nov 17 '16 at 17:44
  • @SanjayManohar, having ortogonal input variables will make it easier for the model to identify their unique effects, but the point here is that all you need to do is fit a multiple logistic regression model. – gung - Reinstate Monica Nov 17 '16 at 17:51
  • Great thank you. Yes, I guess my original question was really about how to identify the unique effects, because the basic multiple logistic regression was not attributing the coefficients as I had intended. So I will go with orthogonalised predictors. – Sanjay Manohar Nov 17 '16 at 18:05
  • 2
    Gung you are not correct in saying that one can completely partial out the effect of one predictor simply by including it in a multiple regression with the other predictor of interest. When the predictors are correlated, the shared variance can be allocated to either predictor in the regression equation without affecting overall prediction. This means the fitted weights do not have a unique interpretation unless the predictors are orthogonal. See this article by JA Jones and NG Waller (2016) "Fungible weights in logistic regression", Psychological Methods, 21(2), 241. There is an R package "fu – Geoff Stuart Apr 02 '19 at 23:36