3

Imagine a dataset where there are 3 nested grouping variables: study $>$ group $>$ outcome (read outcome is nested in group which in turn is nested in study.)

We have two predictors. ktype is a categorical factor (0=direct,1=indirect) that can vary between studys, between groups, and between outcomes. And, treat is a continuous variable only varying between studys.

Question: If I fit 3 models (see below) that only differ in their random-effects specification, then, how does the interpretation or meaning of any of the 3 fixed-effect coefficients (A, B, C) for each model change?

         Estimate
ktype0     A
ktype1     B
treat      C

# Syntax in R's `lme4` package (DON'T RUN)
1) lmer(y ~ 0 + ktype + treat + (ktype |study))

2) lmer(y ~ 0 + ktype + treat + (ktype |study) + (treat |study))

3) lmer(y ~ 0 + ktype + treat + (ktype |study/group/outcome) + (treat |study))

# STRUCTURE OF GROUPING VARIABLES:
"
study  group outcome
1       1    1
1       1    2
1       1    1
1       1    2
1       1    1
1       1    2
1       2    1
1       2    2
1       2    1
1       2    2
1       2    1
1       2    2
2       1    1
2       1    2
2       2    1
2       2    2"
```
rnorouzian
  • 3,056
  • 2
  • 16
  • 40

1 Answers1

1

In my experience, fixed effects are usually well-estimated when the number of clusters is large enough so that the assumption of multivariate normal random effects can be justified, even when the random structure is mis-specified.

When the number of clusters is small, then parameter estimates can be biased. The extent of the bias is hard to know in advance, since it depends very much on the structure of the data and th underlying data generation process. Simulation studies are the best way to approach this, using specific data structures and a range of values for various options:

  • number of levels of each grouping factor
  • correlations among covariates
  • types of covariates
  • values for the fixed effects in the model
  • variance of random intercepts for each grouping variable
  • variance of random slopes where applicable
  • correlations between random intercepts and slopes

The idea would be to simulate data for such a model, and then run a Monte-Carlo simulation to assess whether the fitted values are biased. I have posted code on here which could be easily adapted for this. eg:
Why is this linear mixed model singular?
If I consider the fixed factor as a random slope, the p-value changes from p<0,05 to p>0,05

Robert Long
  • 53,316
  • 10
  • 84
  • 148
  • Dear Robert, thank you so much. I believe my question might not have been clear enough. My real intent was to see if the actual interpretation/meaning of fixed effects coefficients (`A, B, C`) might differ across the 3 models I showed in my question. For example, what is the difference in the interpretation/meaning of `treat` in model 1 vs. model 2 vs. model 3. And what is the difference in the interpretation/meaning of `ktype0` in `model 1` vs. `model 2` vs. `model 3`. My own understanding is that the interpretation/meaning of `treat` in . . . – rnorouzian Aug 12 '21 at 15:05
  • . . .`model 1` is: change in `y` for each unit of increase in `treat` when `ktype == 0`. However, I wonder how this interpretation (or meaning of `treat` as a fixed-effect coefficient) changes when `treat` is itself taken as random-effect as in `model 2` or `model 3`? The same question applies to `ktype`. Given that in `model 1`, `ktype` is only taken as random at the `study` level, and then in `model 3` taken to be random at the `study`, `group`, and `outcome` levels, I wonder how this interpretation (or meaning of `ktype0` as a fixed-effect coefficient) changes when `ktype0` used in . . . – rnorouzian Aug 12 '21 at 15:06
  • . . . `model 1` vs. in `model 3`? – rnorouzian Aug 12 '21 at 15:06
  • As I explained in my answer, it all depends on the structure and details of your actual data as to whether the parameters will be biased. I don't know what you mean by "My real intent was to see if the actual interpretation/meaning of fixed effects coefficients (A, B, C) might differ across the 3 models I showed in my question". The interpretation of a fixed effect in a linear mixed model is exactly the same as a non-mixed model - the estimated change in the response for a unit change in the predictor. This has nothing to do with the random structure (correctly specified or not) – Robert Long Aug 12 '21 at 15:09
  • But Roberts, when we allow say `ktype0` and `ktype1` in `model 3` to vary across, `group`, `outcome`, and `study` levels, won't the fixed effect of these two coefficients be each a coefficient that has been averaged across these three levels? By contrast, wouldn't `ktype0` and `ktype1` in `model 1` that are only allowed to vary across `study` each represent a coefficient that has been averaged across `study` levels? – rnorouzian Aug 12 '21 at 15:17
  • I don't understand your last comment. The fixed effects are the estimated change in the response for a unit change in the predictor, leaving the other fixed effects constant, regardless of the random structure. I don't know what you are asking now. – Robert Long Aug 12 '21 at 15:24
  • Let me ask it differently. For the exact same data, with the exact same fixed effect specification, if we create models that only differ in their random-effect specification, then is it **possible** for the the same fixed-effects coefficients to be **substantially** different in value across these models? If yes, what could that be attributed to? – rnorouzian Aug 12 '21 at 15:28
  • Also: "*My own understanding is that the interpretation/meaning of treat in model 1 is: change in y for each unit of increase in treat when ktype == 0*". Why "when ktype == 0" ? It is not involved in an interaction. – Robert Long Aug 12 '21 at 15:28
  • "*For the exact same data, with the exact same fixed effect specification, if we create models that only differ in their random-effect specification, then is it possible for the the same fixed-effects coefficients to be substantially different in value across these models*". YES ! That's exactly what I am saying in my answer. When the cluster sizes are large enough there shouldn't be a problem, but when they are small the estimates can be quite different (ie biased) – Robert Long Aug 12 '21 at 15:30
  • oh!! You mean if we have enough data that should not theoretically happen but if it does, it simply is the result of bias due to small levels of the grouping variable(s). But Robert when I took a class, we were told fixed effects are estimated (via empirical Bayes) averages of their random counterparts which led me to believe that changing the specification of random part could **legitimately** (i.e., not due to bias) change the underlying meaning of a fixed effect coef. That class also led me to believe if we don't add a random effect for a fixed effect, then it remains the same as OLS coefs. – rnorouzian Aug 12 '21 at 15:36
  • I would be interested in seeing the lectuiure notes or other materials that support those assertions - do you have anything you can share ? It should be fairly easy to construct simulations that I mentioned in my answer. The models you have in the question are a bit complicated to simulate from scratch and that would take some time, but it should be pretty easy to simulate data for something like `Y ~ X + Y + (1 | site) + (X | site:subject)` and then see what happens when the wrong the random structure is specified (with different data structures) – Robert Long Aug 12 '21 at 15:43
  • I do but please don't share them with anyone else. – rnorouzian Aug 12 '21 at 15:43
  • Thanks. You have my email I think. If you are interested in the simulations I just mentioned then perhaps ask a new question about how to investigate mis-specification of random effects on fixed effects estimates (making sure the question is statistical, not programming) then I will see what I can do :) – Robert Long Aug 12 '21 at 15:45
  • For sure, BTW, the reason I asked this question was apply it to multilevel meta-analysis. For example, see about the bottom of p. 26 of [this paper](https://psyarxiv.com/nj28d/). The model is $R_{jk} = Y_0 + Y_1(Age)_{jk} + v_k + u_{jk} + e_{jk}$. So, the author says: *"$Y_1$corresponds to the difference in average effect sizes between cases that differ in age by one year."* Although he also gives the usual interpretation of a fixed effects later. But that interpretation of $Y_1$ had me think if my interpretation of fixed effect of `treat` in my `model 2` should be different from `model 1`. – rnorouzian Aug 12 '21 at 15:55
  • That's the wrong link but I've found it now :) – Robert Long Aug 12 '21 at 16:51
  • So sorry Roberts. [HERE](https://stats.stackexchange.com/q/540038/140365) is the correct link. Also, I found [this answer](https://stats.stackexchange.com/a/478580/140365) as providing some relevant intuition as to why it might be useful to think of a fixed-effect coeff in a mixed model as the overall (weighted) average of individual coeffs from each individual clusters. – rnorouzian Aug 12 '21 at 17:02