7

I have a sample including all students within 52 schools which have been randomly assigned to be treatment or control (we have group randomized trial in which all students in each school is either assigned to treatment or to control ). I'd like to know whether there is any school clustering effect in this data. In other words, I'd like to compare the within-school variability of students to between school variability. One way to measure whether there is any school clustering effect is to use ICC (intra class correlation). The ICC for my data (with schools as clustering units) is 0.04. I'd like to have a statistical method helping me decide whether ICC of 0.04 is large enough to take the school clustering effect into account in my model. What would you recommend? Can F test on the ratio of the variability between schools over the variability with-in schools answer this question?

Thank you for helping me on this issue.

Sam
  • 2,104
  • 19
  • 29
  • How are you estimating the ICC? Does that method give you a $p$-value? – Macro Jul 27 '12 at 01:21
  • Hi @Macro: I simply find the MSbetween and MSE and I calculate MSbetween/(MSE + MSbetween). There is another way and that is to use the variance of the random effect (for school) and the residuals random effect. The answer to your second question is NO; I don't get any p-value. – Sam Jul 27 '12 at 02:14
  • you can estimate the ICC by using a random effects model, as you indicated, although this typically makes an additional assumption of normality. In particular, if $Y_{ij}$ is the outcome for person $i$ in school $j$ then you can fit the model $$Y_{ij} = \mu + \eta_{j} + \varepsilon_{ij}$$ where $\eta_j$ is the school random effect. If ${\rm var}(\eta_j) = 0$, then there is no intra-school correlation, and you can test this with the likelihood ratio test. This would be one way of making this determination – Macro Jul 27 '12 at 11:05
  • I wouldn't use a significance test here. The important thing is not the significance of the ICC, but its size and effect. I would look for literature on how big the ICC had to be to make the multilevel model needed. A practical alternative is to do some analysis with it both as a multilevel model and not, and see if the results are different. If they are, go with the MLM – Peter Flom Jul 27 '12 at 11:09
  • Thank you @Macro. I appreciate your help, you always provide very useful answers. – Sam Jul 27 '12 at 15:46
  • @Macro: I appreciate it. I posted my answer. Would you please check the random effect model. Should I adjust for other covariates that i'm interested? also, should we only look at students' baseline scores in this longitudinal study to calculate ICC? – Sam Jul 27 '12 at 17:22

1 Answers1

4

Goal: To calculate ICC and to test whether the resulted ICC is big enough to account for the clustering effect in the study.

My Study: All students within 52 schools followed longitudinally over three years. Schools are randomly assigned to Treatment or control. My question is do we need to account for school clustering effect?

ICC is a measurement that can help us figuring out whether clustering effect is important or not.

How to calculate ICC?

1) Looking at students scores in the baseline year where no one used the treatment and calculating: $$MS_{between} / (MS_{between} + MS_{Err})$$

2) Another way is to calculate ICC by using random effects model framework and by use the variance of the random effect (for school). This is what is most commonly done with varying group sizes.In this case, ICC is $$\sigma_b^2 / (\sigma_b^2 + \sigma_e^2)$$ where $\sigma_b^2$ is the variance of the random effect and $\sigma_e^2$ is the residual variance.

How to fit the random effect model in R?

library(nlme)
schRandModel <- lme(StudScoresBaseline ~ 1, data = data, random = ~ 1 | SCHOOL)
SchRandModel.LogLiklihood <- schRandModel$logLik

The model above assumes random intercepts for the schools. Now we fit the model without schools as random effect (groups are students rather than schools):

simpleModel <- lme(StudScoresBaseline ~ 1, data = data, random = ~ 1 | StudentID)
simpleModel.LogLiklihood <- simpleModel$logLik

Finally, using the likelihood ratio test, we test the difference between likelihoods:

logLikDiff = SchRandModel.LogLiklihood-simpleModel.LogLiklihood
1-pchisq(2*logLikDiff,df=1)

my resulted p-value is less than <0.0001 indicating that we have enough evidence that random effect model is a better fit. Therefore, school clustering effect is important in this study .

I'd like to thank Macro for guiding me step by step to write the answer.

Sam
  • 2,104
  • 19
  • 29