3

I am currently running a three-level meta-analysis, due to the fact that I have multiple dependent effect sizes in many studies. I do not know the actual dependencies between effect sizes, however, so when there are dependencies, they are assumed to be $\tau^2_{(3)}$, per Cheung (2015, Ch. 6) and Konstantopoulos (2011). I am using the rma.mv() function in the metafor package to do this and employing a meta-regression approach, such that my effect sizes es (in my meta-analysis, Fisher's Z) are predicted by two moderators, pubyear and subtle. The model looks like this:

mod2.1 <- rma.mv(es ~ pubyear + subtle, 
                 random = ~1|studyid/effectid, 
                 V = var, data = meta)

My main interest is in pubyear. I find that, after adding subtle as an additional covariate, the effect of pubyear gets smaller.

This leads me to ask the question: Is subtle predicted by pubyear? I do not know how I would estimate this, so my question is: How can I model the relationship between two moderators in a meta-regression?

Right now, my meta-analysis shows that the es gets smaller as pubyear increases, so now I want to know: Has the subtlety of measures (subtle) also increased over time (pubyear)? How would I do this?

The intuitive solution would to simply correlate the two with one another. But this does not take into account that one study might report 8 effect sizes, while another study may only report 1. Obviously, the study with 8 effect sizes will have more weight. I plotted this relationship, regardless. You can see that there may be a relationship when doing this weirdly-weighted regression (lm line in blue, loess line in red):

enter image description here

Has anyone run into this before or know how to handle it?

mdewey
  • 16,541
  • 22
  • 30
  • 57
Mark White
  • 8,712
  • 4
  • 23
  • 61
  • 1
    I'll take a look more in-depth later in the week if I can find the time, but in the meantime, Lipsey (2003; https://www.jstor.org/stable/1049948?seq=1#page_scan_tab_contents) and this CV thread here (https://stats.stackexchange.com/questions/262326/variance-inflation-factor-for-meta-analyses) might be able to get you started. – jsakaluk May 27 '17 at 22:12
  • Brilliant, thanks. I didn't think about searching for VIF. – Mark White May 27 '17 at 22:14
  • 1
    Also worth looking at the variance covariance matrix of your coefficients which is sometimes enlightening. – mdewey May 29 '17 at 13:12

1 Answers1

2

So, if I understand things correctly, subtle is a variable at the effect size level and pubyear is a variable at the study level. If you really want to examine if subtle is predicted by pubyear, then this could be done using a standard multilevel model (with random intercepts). Using the nlme package, you can fit this with:

res <- lme(subtle ~ pubyear, random = ~ 1 | studyid, data=meta)
summary(res)

Of course, this assumes a linear relationship, but more complex relationships can also be modeled (but I wouldn't use that loess line as any indication, as that seems like a total overfit in my opinion).

Wolfgang
  • 15,542
  • 1
  • 47
  • 74
  • 1
    Yup, you got it. Thanks, I thought about doing this, too, but wasn't sure. I also thought about doing a WLS where the weights for each data point would be $1/k_j$, where $k$ is the number of effect sizes in study $j$. This would control for the problem that some studies report 15 effect sizes, while others report 1, leading an OLS to be biased. But I think I'll fit an `lmer` object, because it doesn't ignore the dependencies between the effect sizes within a study. – Mark White May 29 '17 at 21:08