ICC in a multi level model with two random effects

Question

My understanding is that intraclass correlation gives you an idea of how much variance your level two factor can explain in overall variance of the dependent variable. It is supposed to give an indication of whether a multi level analysis is justified by the variance structure of the data.

Now, I specified a null model, as a starting point for a mixed effects analysis where items of a test are nested within subjects. So I treat each subject ID as a level two unit grouping 28 items. Looks like this:

Mt01 <- lmer(APMt ~ (1|ID) + (1|content), data=mlmData, REML=FALSE)

#Output:
Random effects:
 Groups   Name        Variance Std.Dev.
 ID       (Intercept) 0.102    0.319   
 content  (Intercept) 0.300    0.548   
 Residual             0.213    0.462   
Number of obs: 2940, groups:  ID, 105; content, 28

The dependent variable is log(time) for each item. I specified two random effects, one due to subjects and one due to item content (all subjects worked on the same items).

How would I compute the ICC? Do I compute one for each random factor? Is it even necessary in this scenario? How exactly would I interpret the value?

Here are some of my approaches:

InterceptVarSubj <- VarCorr( Mt01 )$ID
    InterceptVarItem <- VarCorr( Mt01 )$content
ResidualVar <- attr( VarCorr( Mt01 ), "sc")^2

#A
icc <- InterceptVarItem / (InterceptVarItem + ResidualVar)
icc
[1] 0.584
#B
icc <- InterceptVarItem / (InterceptVarItem + InterceptVarSubj + ResidualVar)
icc
[1] 0.488
#C
icc <- InterceptVarSubj / (InterceptVarSubj + ResidualVar)
icc
[1] 0.323
#D
icc <- InterceptVarSubj / (InterceptVarItem + InterceptVarSubj + ResidualVar)
icc
[1] 0.165

Answer 1

There are several ways to calculate and interpret the ICC for a mixed model. A useful thread is here. To calculate, it is the amount of variance from certain factor(s) divided by the total variance. That would be akin to your B and D calculations. This can be interpreted as the correlation of two randomly drawn units from the same grouping or as the amount of variance explained by those groupings, similar to an $R^2$.

This calculation (B): $$\frac{\sigma^{2}_{item}}{\sigma^{2}_{subj}+\sigma^{2}_{item}+\sigma^{2}_{res}}$$ could be interpreted as the correlation of scores for any item, regardless of the subject using the item.

Similarly, this calculation (D): $$\frac{\sigma^{2}_{subj}}{\sigma^{2}_{subj}+\sigma^{2}_{item}+\sigma^{2}_{res}}$$ could be interpreted as the correlation of scores from any subject, regardless of what item they are working on.

A combined calculation, such as this: $$\frac{\sigma^{2}_{subj}+\sigma^{2}_{item}}{\sigma^{2}_{subj}+\sigma^{2}_{item}+\sigma^{2}_{res}}$$ could be interpreted as the correlation between scores of the same person using the same item (in your data, this is 0.654).

Turning to the values you listed (0.488 and 0.165, respectively), a naive interpretation is the items you're using are modestly correlated regardless of who is using it, while the subject scores are mostly uncorrelated across all the items.

Finally, regarding your two additional calculations (A and C), I'm not aware of a useful interpretation of those values, however I have an open question on the topic.