Inconsistent delta.chisq.scaled using Lavaan and semTools?

Question

I've recently been using Lavaan and semTools to test for measurement invariance in confirmatory factor analysis (CFA) models and I noticed an (apparent) inconsistency in a calculation that confused me. When you use the "measurementInvariance" command from semTools to test nested models (using MLR), the chi-square difference value (delta.chisq.scaled) is not the same as when you manually extract the fit statistic from each model and subtract them yourself. It also does not match the difference in the non-scaled chi-square values. Am I missing something about how this is calculated?

I've given an example below using demo data from Lavaan. Thanks in-advance!

require(lavaan)
require(semTools)
HW.model <- 
 'visual =~ x1 + x2 + x3
 textual =~ x4 + x5 + x6
 speed =~ x7 + x8 + x9
 '
out<-measurementInvariance(HW.model, data=HolzingerSwineford1939, estimator = "MLR",group="school")
modelDiff<-compareFit(out)
summary(modelDiff, fit.measures="all")

The resulting values are:

1. measurementinvariance command chi-square (scaled) diff = 6.567
2. manual subtraction of extracted chi-square (scaled) values = 4.25
3. manual subtraction of extracted non-scaled chi-square values = 7.68

Does anyone know why the values from list item 1 & 2 above differ?

Answer 1

Thanks to Jeremy Miles for leading me to the key. I needed to divide the scaled chi-square difference by the scaling correction. As is described on statsmodel.com:

Following are the steps needed to compute a chi-square difference test in Mplus using the MLM (Satorra-Bentler), MLR, and WLSM chi-square. DIFFTEST should be used for MLMV and WLSMV. The nested model is the more restrictive model with more degrees of freedom than the comparison model.

Compute the difference test scaling correction cd, where d0 is the degrees of freedom in the nested model, c0 is the scaling correction factor for the nested model, d1 is the degrees of freedom in the comparison model, and c1 is the scaling correction factor for the comparison model. Be sure to use the correction factor given in the output for the H0 model. cd = (d0 * c0 - d1*c1)/(d0 - d1) Compute the Satorra-Bentler scaled chi-square difference test TRd as follows: TRd = (T0*c0 - T1*c1)/cd where T0 and T1 are the MLM, MLR, or WLSM chi-square values for the nested and comparison model, respectively. For MLM and MLR the products T0*c0 and T1*c1 are the same as the corresponding ML chi-square values.

UPDATE (6/18/14): It appears that in lavaan you can now simply specify "MLM" as your estimator and then it will calculate the SB chi-square difference for you when you do a comparison using ANOVA.

For Example:

require(lavaan)

HS.model <-  'visual  =~ x1 + x2 + x3
                  textual =~ x4 + x5 + x6
                  speed   =~ x7 + x8 + x9 
                  visual + textual + speed ~ grade
                 textual + speed ~ visual'
fit <- cfa(HS.model, data=HolzingerSwineford1939, estimator="MLM", group="sex")
fitb <- cfa(HS.model, data=HolzingerSwineford1939, estimator="MLM", group="sex", group.equal="regressions")
anova(fit, fitb)

You can extract/calculate other scaled fit statistic differences to report by doing the following:

T1<-fitMeasures(fit)[c("cfi.scaled", "rmsea.scaled", "aic")]
T2<-fitMeasures(fit.b)[c("cfi.scaled", "rmsea.scaled", "aic")]
x<-round(T2-T1, digits=3)

NOTE: The AIC is not scaled and doesn't change when MLM is used, but I like to get all my stats to report together, so that is why it is in the code I provided.