2

I am modeling a continuous dependent variable with a couple of covariates (known a priori) and a variable of interest.

I ran into an issue of interpretation which I'd like to clear up. When I include a categorical predictor (Sex) my variable of interest is no longer significant. However, when I run two separate models (one for males, one for females) the variable of interest is only significant in females.

In the full model (with sex as an IV) the interaction between variable of interest * sex is non-significant.

How do I proceed? How do I know for sure I have justification to run two separate GLMs based on sex? I tried plotting the the variable of interest by sex vs. my dependent variable and observed no significant difference in the slopes (although visually these were strikingly different and the lines intersected) and significant difference in intercept.

Frans Rodenburg
  • 10,376
  • 2
  • 25
  • 58
mindhabits
  • 307
  • 2
  • 12
  • Thanks for your response and help in the past. I will take your advice. I still have a lingering question: Let's say I have good reason to consider that there is a sexual dimorphism in my variable of interest (due to influence of sex hormones etc.). Would the chain of logic have held If the slopes were significantly different when covariate of interest are plotted against dependent variable (or its biggest determinant) grouped by sex? Meaning, if significantly different than I would have "statistical backing" to proceed with 2 separate GLMs (including underlying biological theory) – mindhabits Jul 03 '19 at 04:15
  • I ran into an issue of interpretation which I'd like to clear up. When I include a categorical predictor (Sex) my variable of interest is no longer significant. However, when I run two separate models (one for males, one for females) the variable of interest is only significant in females. THis simply suggests female factor operates as a moderator. –  Jul 03 '19 at 08:19
  • In the full model (with sex as an IV) the interaction between variable of interest*sex is non-significant. You are applying Anova model here. The Anova is based on normality assumption. The sum total of deviations is zero. And u get a nonsignificant test statistic. –  Jul 03 '19 at 08:30
  • relevant portion of your comments may be considered for inclusion in the Question to improve the quality of this question. –  Jul 04 '19 at 08:20

1 Answers1

0

Model selection based on $p$-values will bias the coefficients of the final model towards significance. This is a form of stepwise regression and should be avoided if the goal is confirmation through $p$-values. The actual chance of a false positive will be much higher than the chosen level of significance. This has been discussed in several places on CV, most notably here.

You should therefore go with the model you originally intended to use for inference, which I'm assuming is the full model with interaction.

Even (or especially) if you have expert knowledge suggesting that gender affects the variable of interest, a single model using an interaction with gender is preferred. The interaction allows you to model gender-specific effects while having a single model has multiple advantages (most importantly: more observations).

Note that just because the effect isn't significant doesn't mean it isn't there. You just couldn't demonstrate it. This might be because (1) your sample size is limited compared to the number of parameters, (2) the effect size is small, (3) the variance of the effect is large, or a combination of these reasons. For this reason, model selection based on $p$-values usually isn't very helpful.

Frans Rodenburg
  • 10,376
  • 2
  • 25
  • 58
  • most importantly: more observations). This needs to be substantiated for it seems illogical to be considered for selecttion of a Model. –  Jul 03 '19 at 23:36
  • Under the assumption that the data generating processes for male and female share any kind of similarity (e.g. intercept, slopes of other covariates, error distribution), more observations means more degrees of freedom, better estimates of your other covariates, easier checking of (and possibly even meeting) distributional assumptions. Also see [here](https://stats.stackexchange.com/a/126501/176202), [here](https://stats.stackexchange.com/a/332965/176202), and [here](https://stats.stackexchange.com/q/412962/176202). – Frans Rodenburg Jul 03 '19 at 23:49
  • I am not sure whether combination (single model) can yield better results. Your focus is on similarity while OP is seemingly trying check the effect of sex or gender -type on a specific variable(continuos). I can makeout much from the question becausei it is somehow vague e.g. variable of interest is not defined to be meaningful or contextual.. –  Jul 04 '19 at 03:59
  • I appreciate your commenting on my answer, but I do not understand your point of contention. Even if male and female have different *slopes* for the explanatory variable of interest, they still (probably) share the same error structure. Anything else they share further increases the efficiency of using a single model vs multiple models. I hope you'll agree that splitting your data reduces the sample size, and that larger sample sizes tend to result in better parameter estimates. – Frans Rodenburg Jul 04 '19 at 04:18
  • Model selection based on p-values will bias the coefficients of the final model towards significance. It is not clear to me. None-the-less Your Ans talks about step regression while OP is talking GLM –  Jul 04 '19 at 08:38
  • Would you please differentiate the step regression from the GLM and linear models. –  Jul 04 '19 at 08:55
  • I'll address your comments in order: (1) The question is which of two modelling approaches to use. If that dilemma arose from looking at the $p$-values of coefficients, then that is stepwise regression. The problems with stepwise regression have been talked about numerous times here ([1](https://stats.stackexchange.com/a/17594/176202), [2](https://stats.stackexchange.com/a/20856/176202), [3](https://stats.stackexchange.com/a/218223/176202)). (2) The model happens to be a GLM, but the question is which model to choose. – Frans Rodenburg Jul 04 '19 at 09:19
  • Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/95710/discussion-between-subhash-c-davar-and-frans-rodenburg). –  Jul 04 '19 at 10:39
  • @SubhashC.Davar I tried my best to answer your questions, but apparently it is not working, sorry. You are free to post a question asking further clarification on the subject, which is more likely to attract answers from others, who may better understand your concerns. Respectfully, I have no desire to further discuss this in chat. – Frans Rodenburg Jul 04 '19 at 10:57