1

I have two "unrelated" data frames (df1, and df2). The type and number of columns and rows is identical, just the actual data differs.

I am fitting the same base and full models over both:

df1_base_model = lmer(Time~1         + (1|Question) + (1|Participant), df1, REML=FALSE)
df1_full_model = lmer(Time~Condition + (1|Question) + (1|Participant), df1, REML=FALSE)
df2_base_model = lmer(Time~1         + (1|Question) + (1|Participant), df2, REML=FALSE)
df2_full_model = lmer(Time~Condition + (1|Question) + (1|Participant), df2, REML=FALSE)

To see whether the introduction of Condition as a fixed factor makes any difference, I run:

anova(df1_base_model, df1_full_model)
anova(df2_base_model, df2_full_model)

and get this output for df1:

               Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)  
df1_base_model  4 2028.2 2045.0 -1010.1   2020.2                           
df1_full_model  6 2024.6 2049.9 -1006.3   2012.6 7.5162      2    0.02333 *

and for df2:

               Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)    
df2_base_model  4 1402.0 1418.9 -697.02   1394.0                             
df2_full_model  6 1280.9 1306.1 -634.45   1268.9 125.14      2  < 2.2e-16 ***

According to the output, the introduction of Condition is statistically significant for both df1 and df2. But I need some way to quantify the effect, beyond just a p value. The data is such that the introduction of Condition has only a marginal effect in df1 but a very large effect in df2, and the difference of p values already suggest that, but I need a better effect size measure.

So my question is: What is the proper way to compute and compare effect sizes in this situation? I've read that many traditional measures, such as Cohen's d or (partial/generalized) eta squared do not apply to linear mixed models with random effects. So what ways are there to compare the models for df1 and df2 and show that the effect size of introducing Condition is much larger for df2?

EDIT: In case it helps, my output variable Time is measured in seconds. Perhaps there is a way to express the effect size in terms of seconds somehow?

Dimitar Asenov
  • 143
  • 6

1 Answers1

1

You can just combine the two data sets and add appropriate interaction terms between dataset and the other variables. Then you will get an effect of Condition and an effect of dataset:Condition so that you can evaluate if the difference in effect of Condition is different between the data sets.

JonB
  • 2,658
  • 1
  • 9
  • 22
  • I have tried the analysis in the way you suggest, but it gets tricky, as in reality I have 3 data sets not 2. But even if I put them together and add an interaction term, my question still remains. lmer will output that the interactions are significant, but how do I get the effect size for both cases? – Dimitar Asenov Sep 09 '15 at 15:42
  • If you have three datasets called df1,df2 and df3 and you add the interaction terms, your output will display a coefficient for Condition, a coefficient for df2:Condition, and a coefficient for df3:Condition. You can report that the effect of Condition for df1 was the coefficient for Condition, and for df2 you have Condition + df2:Condition, and for df3 you have Condition + df3:Condition. The p-values for df2:Condition and df3:Condition will indicate a significant difference vs df1. A problem is in calculating the confidence intervals. There is a formula for this, but I don't know it by heart. – JonB Sep 09 '15 at 15:58
  • Now that I investigated this, it is beginning to make sense. Two further questions though: (1) If I just look at the coefficients, they provide the difference to the respective means of df1, df2, df3. This difference in itself is not very useful, since it doesn't tell me what percentage of the overall mean that is. Is it OK to just divide this by the group mean? And what are the group means anyway? I think the intercept is the mean for df1, but what about the rest? Is it intercept + df2 coefficient? (2)I'm a bit sceptical reviews will like this. Is there a more standard approach? – Dimitar Asenov Sep 09 '15 at 18:37
  • The intercept should be the mean of df1 (without Condition). Intercept + Condition is the mean for df1 with Condition. Intercept + df2 is the mean of df2, and Intercept + df2 + Condition + df2:Condition is the mean of df2 with Condition. And the same goes for df3 but replace df2 with df3 of course. I think calculation of the means this way, and also percentages, are uncontroversial, but the problem is how to calculate the confidence intervals. I wouldn't hesitate to use this kind of analysis in my field and I don't think reviewers would dislike it. – JonB Sep 10 '15 at 06:25
  • Thanks, that's what I suspected. Could you please point me to an example or two of research articles where this is done? I'm especially interested in papers that report percentages in this way. – Dimitar Asenov Sep 10 '15 at 08:03