37

I have the following output: 

Generalized linear mixed model fit by the Laplace approximation 
Formula: aph.remain ~ sMFS2 +sAG2 +sSHDI2 +sbare +season +crop +(1|landscape) 

 AIC   BIC    logLik deviance
 4062  4093  -2022   4044

Random effects:
Groups    Name        Variance Std.Dev.
landscape (Intercept) 0.82453  0.90804 
Number of obs: 239, groups: landscape, 45

Fixed effects:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept)  2.65120    0.14051  18.868   <2e-16     
sMFS2        0.26922    0.17594   1.530   0.1260    
sAG2         0.09268    0.14529   0.638   0.5235    
sSHDI2       0.28345    0.17177   1.650   0.0989  
sbare        0.41388    0.02976  13.907   <2e-16 
seasonlate  -0.50165    0.02729 -18.384   <2e-16 
cropforage   0.79000    0.06724  11.748   <2e-16 
cropsoy      0.76507    0.04920  15.551   <2e-16 

Correlation of Fixed Effects:
           (Intr) sMFS2  sAG2   sSHDI2 sbare  sesnlt crpfrg
sMFS2      -0.016                                          
sAG2        0.006 -0.342                                   
sSHDI2     -0.025  0.588 -0.169                            
sbare      -0.113 -0.002  0.010  0.004                     
seasonlate -0.034  0.005 -0.004  0.001 -0.283              
cropforage -0.161 -0.005  0.012 -0.004  0.791 -0.231       
cropsoy    -0.175 -0.022  0.013  0.013  0.404 -0.164  0.557

All of my continuous variables (denoted by a small s before the variable name) are standardized (z-scores). season is a categorical variable with 2 levels (early and late), and crop is a categorical variable with 3 levels (corn, forage, and soy).

This correlation of fixed effects matrix is really confusing me, because all of the correlations have the opposite sign that they do when I look at the simple regressions of pairs of variables. i.e., the correlation of fixed effects matrix suggests a strong positive correlation between cropforage and sbare, when in fact there is a very strong NEGATIVE correlation between these variables - forage crops tended to have much less bare ground compared to corn and soy crops. Pairs of continuous variables have the same issue, the correlation of fixed effects matrix says everything is the opposite of what it should be ... Could this just be due to the complexity of the model (not being a simple regression)? Could it have something to do with the fact that the variables are standardized?

Thanks.

usεr11852
  • 33,608
  • 2
  • 75
  • 117
susie
  • 641
  • 2
  • 8
  • 9

3 Answers3

36

The "correlation of fixed effects" output doesn't have the intuitive meaning that most would ascribe to it. Specifically, is not about the correlation of the variables (as OP notes). It is in fact about the expected correlation of the regression coefficients. Although this may speak to multicollinearity it does not necessarily. In this case it is telling you that if you did the experiment again and it so happened that the coefficient for cropforage got smaller, it is likely that so too will would the coeffienct of sbare.

In part his book "Analyzing Linguistic Data: A Practical Introduction to Statistics using R " dealing with lme4 Baayen suppresses that part of the output and declares it useful only in special cases. Here is a listserv message where Bates himself describes how to interpret that part of the output:

It is an approximate correlation of the estimator of the fixed effects. (I include the word "approximate" because I should but in this case the approximation is very good.) I'm not sure how to explain it better than that. Suppose that you took an MCMC sample from the parameters in the model, then you would expect the sample of the fixed-effects parameters to display a correlation structure like this matrix.

russellpierce
  • 17,079
  • 16
  • 67
  • 98
  • 4
    I'm sorry, this will probably be a silly question, but then why is it important to consider that correlation? I mean, in which situations should that output be considered? – mtao Mar 30 '17 at 17:41
  • 1
    @Teresa It depends on what you are using it for. If you care about the interpretation, then it is telling you about how confusable two sources of effect are. If you care about prediction, it tells you a bit about what other prediction models might look like and gives you some hint as to how the model might change if you dropped a predictors. – russellpierce Apr 01 '17 at 15:11
  • 1
    So, imagine that I have two variables in that output with a correlation of 0.90, for instance. In terms of interpretation, I assume I should drop one of them, because they are "confusable" and seem to be telling the same information. As for prediction, if I drop one of them, other models shouldn't change that much, as they are correlated, am I right? Or am I interpreting this wrongly? – mtao Apr 05 '17 at 07:48
  • 6
    You know, I think you're echoing what I said correctly; but, on reflection I'm not 100pct sure I'm right. You may be best served by opening a new question - that'll get more eyes on your question and increase the likelihood of you receiving a correct answer. – russellpierce Apr 05 '17 at 11:08
  • 1
    @russellpierce, thanks for this answer. One question though, I learned that multicollinearity occurs when _predictors_ correlate with each other. But in your answer you say that it is the correlation of the _regression coefficients_ (not of the predictors) that may speak to multicollinearity. Why not just correlating the predictors themselves instead of the estimated coefficients? – locus Nov 15 '18 at 23:50
  • Like I said above, I'd encourage a new question on these details. My uncertain claim above is that correlation of random effects may be related to multicollinearity, but that they are distinguishable properties. As to why this is the diagnostic info provided by lme4, we'd have to ask the authors – russellpierce Nov 16 '18 at 01:18
  • I suspect it is because these correlations are relevant to model specification. In particular one that comes up in nature a lot is a correlation between the intercept and a slope. Indicating the effect of the slope correlates with the effect of the intercept. There isn't a predictor to stand in for the intercept. So, for this example we can see that this view of the data isn't exactly the same as a question of collinearity of predictors. – russellpierce Nov 16 '18 at 01:23
  • 1
    There seems to be a gap of references (knowledge?) as to why the correlations of fixed effect models appear in the summary and why they are useful: GLMMs arise as a generalisation of GLMs to model heterogeneity/overdispersion as a result of dependencies between predictors (think of, e.g., longitudinal data). In a first analysis of your data you may not spot out the latter and, following Occam's Razor, you may fit a GLM to start with. A way to diagnose whether there are dependencies is to compute the correlation matrix of the resulting coefficient estimates. – Coca Feb 15 '19 at 11:32
  • 2
    Continued: if there are relatively high correlations you may fit a GLMM, and the way to check whether it (or, more precisely, its random effects) satisfactorily modelled the dependencies is by computing the correlation matrix of the fixed effect models and comparing it to the one from the GLM. When GLMM addresses the issue of dependencies (e.g., longitudinal data), you will see a dramatic decrease in the correlations. – Coca Feb 15 '19 at 11:33
  • 1
    Given the general lack of references on this (worse off, some people in charge of code, see https://github.com/lme4/lme4/issues/300, want to remove the correlation matrix! I would say one should add the one from GLM...) and following @russellpierce's suggestion, it would be a good idea to create an entry for this. – Coca Feb 15 '19 at 11:33
1

It can be helpful to show that those correlations between fixed effects are obtained by converting the model's "vcov" to a correlation matrix. If fit is your fitted lme4 model, then

vc <- vcov(fit)

# diagonal matrix of standard deviations associated with vcov
S <- sqrt(diag(diag(vc), nrow(vc), nrow(vc)))

# convert vc to a correlation matrix
solve(S) %*% vc %*% solve(S)

and the correlations between fixed effects are the off-diagonal entries.

Ed Merkle
  • 66
  • 2
0

If your negative and positive correlations are the same in their value and only their sign differ, you are entering the variable mistakenly. But I don't think this is the case for you as you already seem quite advanced in stats.

The inconsistency you are experiencing can be and is likely caused by multicollinearity. It means when some independent variables share some overlapped effects, or in other words are correlated themselves. for example modeling to variables "growth rate" and "tumor size" can cause multicollinearity, as it is possible and likely that larger tumors have higher growth rates (before they are detected) per se. This can confuse the model. And if your model has few independent variables which are correlated with each other, interpreting the results can sometimes become quite difficult. It sometimes leads to totally strange coefficients, even to such extents that the sign of some of the correlations reverses.

You should first detect the sources of multicollinearity and deal with them and then rerun your analysis.

Vic
  • 1,126
  • 5
  • 17
  • 28
  • Thanks for responding Vic. When you say 'deal with' the sources of multicollinearity, what do you mean? Just, find out which predictors are highly correlated and choose one of them to keep in the model and toss the other? Also, detecting the sources of multicollinearity should technically just be a matter of looking at the correlation of fixed effects matrix in the output, right? As far as I've read, that's what that part of the output is for. Though I still don't get why it is systematically telling me that the correlations are the opposite of what they are. – susie Apr 25 '13 at 17:47
  • No you should treat them using some protocols. One is centering your variables which means subtracting them from their means. Another way is to find the sources of multicollinearity and exclude them from your model (and if they are really necessary, assess them in other models from which the paired (correlated) variables are dropped) etc. – Vic Apr 25 '13 at 18:54
  • For detecting them, there are some suggested methods. The most acceptable one is to calculate VIF. Variables with VIFs greater than 10 are very likely to be causing multicollinearity. Another way is to check the correlation matrix between your independent variables. Variables with at least 0.4 correlation with any other independent variable are likely sources of multicollinearity. – Vic Apr 25 '13 at 18:56
  • There is yet another possibility. Sometimes they might be really what they are showing. Sometimes it is "we" who should change our theory based on our new validated findings. But if the multicollinearity is the culprit, the reason why the variables have been reversed is that it is possible for a variable to obtain any value, which includes positive and negative values from case to case. – Vic Apr 25 '13 at 18:59
  • I will look into calculating VIF, thanks for that suggestion. I believe my variables are centered already as they are standardized (x-mean)/stdev. The correlation matrix between my independent variables is (I think) the very thing that my question/confusion was about - the matrix at the bottom of my output, 'Correlation of Fixed Effects'. The fixed effects are all my independent variables. (except the random effect, which is basically just a grouping factor). I don't doubt that multicollinearity may be an issue but I'm not sure I understand how it relates to my initial problem. – susie Apr 25 '13 at 19:11
  • You are very welcome. If some of your correlations are above 0.4 then you are in trouble! No, I was kidding, it is treatable, but really annoys before being totally gotten rid of. I see you have even 0.7 or 0.5 correlations which indicate multicollinearity. Finding why multicollinearity causes a specific artifact effect in each case can be a real pain and its effect differs from case to case, depending the correlated variables, their effects and their correlation direction, etc.... So when I personally see multicollinearity, I never care about "why it caused this or that... I just accepted it! – Vic Apr 25 '13 at 19:27
  • Tracing it back is really really difficult or maybe impossible (practically) sometimes, when the model is large and complicated. But one thing is clear: it makes variable coefficients jump! But from what to what? and why? The latter is so impractical to answer. – Vic Apr 25 '13 at 19:29
  • 1
    -1; misleading. OP didn't enter his variables incorrectly and multicollinearity may not be an issue. A correlation between the raw fixed effects might speak to this point, but Simpson's paradox may allow that approach to lead you in the wrong direction. – russellpierce May 26 '13 at 04:44
  • 1
    Why "misleading"? Which part was misleading? I talked very clearly and avoided inferring clear conclusions. What I said is indeed one of multicollinearity signs and tells us we should check VIFs too. But I don't understand how you know or are sure the "OP didn't enter his variables incorrectly and multicollinearity may not be an issue."? – Vic May 26 '13 at 17:19
  • 1
    Besides you haven't even read my post completely (and downvoted it and call it misleading). If you had, you had seen that I have suggested that the OP should check VIFs (as official indicators for multiC) to make sure whether those high correlations are really pointing to MC or not? but anyways, I am open to learning as long as it is free of arrogance and personal attacks. – Vic May 26 '13 at 17:19
  • 1
    @Vic: Didn't see your comments until just now. I didn't mean for you to view my response as a personal attack. I was of the opinion it was misleading and I provided what I believe to be the correct answer above. I read your post in its entirety at the time. I do not know whether I dug into the comments or not. I stand by my downvote. – russellpierce Jul 06 '14 at 19:37
  • 1
    ... but I do allow that I may be mistaken in that judgement. However, it seemed better to explain why I downvoted rather than to just downvote. – russellpierce Jul 06 '14 at 19:38