1

Neglecting sample size, is there something that I can miss if I choose to model separately each of the levels of my categorical variable?

To be more specific, I want to predict a binary outcome $Y$ with two predictors $X$ (continuous) and $G$ (categorical, $n$ levels). I could build one logistic regression model $Y \sim X + G$ or build $n$ regressions $Y \sim X$, one for each level of $G$.

What is the risk of not pooling the data in this latter scenario?

EDIT: I forgot to mention that I have good reasons to think that pooled data is not homogeneous and it's much more homogeneous inside each group.

Patrick
  • 393
  • 1
  • 9
  • You will have smaller samples, hence fewer degrees of freedom in each model, leading to poorer estimates with greater standard errors. You will also have to perform $n\times$ the usual diagnostics. – Frans Rodenburg Apr 29 '19 at 14:46
  • Thanks @FransRodenburg. However, I said "neglecting sample size" because I want to know if there is something *else* than poorer estimates and $n$X analyses that I could miss doing $n$ regressions – Patrick Apr 29 '19 at 14:50
  • Another thing that comes to mind is that you will not be able to directly compare the effect sizes of each category, since they were fitted on different data. You are also more prone to overfit, unless you center each subset and estimate each model without an intercept. Why is sample size not compelling enough, though? Let's say you want to assess the predictive performance of your model. You will now need to split not just your original data, but each of the $n$ subsets into smaller subsets for validation. – Frans Rodenburg Apr 29 '19 at 15:20
  • Sorry @FransRodenburg, I don't understand your overfit point. Why am I more prone to it unless I center each subset? – Patrick Apr 29 '19 at 15:28
  • 1
    Have you not considered multi-level (hierarchical, mixed-effect) models? They provide a principled middle ground between pooled and independent models. – Wayne Apr 29 '19 at 22:13

1 Answers1

2

The general principles behind preferring one overall model are explained quite nicely here. It's not just a matter of smaller $N$ with the separate analyses giving larger X-coefficient standard errors. You also typically will get variability in X-coefficient standard errors among the separate analyses, while the combined analysis takes information from all samples to get more reliable pooled error estimates. Also, if you are interested in whether the relationship of X to outcome differs among the levels of G (as you probably should be), then the pooled model would give a direct test of that possibility with an X:G interaction term in the pooled model. Attempts to pool results of separate analyses to examine interaction would be much more cumbersome and less reliable than a simple direct interaction test based on all the data.

EdM
  • 57,766
  • 7
  • 66
  • 187
  • Thanks @EdM. But is true to say that by not pooling each sub model can show a different slope on X which is not possible by pooling (each group can add only its own intercept to the global one)? – Patrick Apr 29 '19 at 19:54
  • 1
    I realize that by adding an interaction term between $X$ and $G$, the model can have this kind of expressivity. – Patrick Apr 29 '19 at 20:17
  • @Patrick yes, the interaction term allows for different slopes. And the combined analysis will typically provide a more reliable pooled error estimate for determining whether the slopes differ among levels of $G$, which is included in testing for significance of the interaction. – EdM Apr 29 '19 at 20:28
  • I understand now that pooled data gives more reliable estimates. The only annoying thing is that I get slightly better results with non-pooled data. For example, my recall@k measure is 1% better for non pooled vs pooled (with an interaction term). It can seem negligible but it's not: it represents around 20 more positives cases that are recalled by the non pooled predictive model. – Patrick Apr 29 '19 at 20:38
  • 1
    @Patrick this might be where overfitting comes in: you might fit this data set better but not generalize well to a new sample. Also, logistic regression [isn't a classifier](https://stats.stackexchange.com/q/405041/28500). It gives class probabilities. For measures like recall there needs to be a choice of cutoff. See [this question](https://stats.stackexchange.com/q/312780/28500) for why accuracy, recall, etc aren't good metrics for evaluating a logistic model. Get the model first, then choose the cutoff for classification based on your tradeoffs between false positives and negatives. – EdM Apr 29 '19 at 22:40