3

I am using the CausalImpact package in R to calculate the impact of a marketing intervention using Bayesian Structural Time Series. This methodology and package is explained in Broderson et al. 2015 found at https://research.google.com/pubs/pub41854.html (direct PDF link here).

I calculated the impact of a marketing intervention in 18 clinics (the "test" subjects). Using the package, I calculated the causal impact of the intervention compared with the respective synthetic control in each of the 18 clinics.

A replicable example of one sample intervention is available from Google's Github page on CausalImpact and the code is provided below:

library(CausalImpact)

#code from Google's Github page on CausalImpact package
set.seed(1)
x1 <- 100 + arima.sim(model = list(ar = 0.999), n = 100)
y <- 1.2 * x1 + rnorm(100)
y[71:100] <- y[71:100] + 10
data <- cbind(y, x1)

pre.period <- c(1, 70)
post.period <- c(71, 100)
impact <- CausalImpact(data, pre.period, post.period)
summary(impact)

For this example, one can pull out the "Relative Effect" (effect size), along with the lower 95% CI, upper 95% CI, and SD values for this one example intervention.

impact$summary$RelEffect[1]
impact$summary$RelEffect.lower[1]
impact$summary$RelEffect.upper[1]
impact$summary$RelEffect.sd[1]

I have done this for 18 different "test" clinics. For each one, I have the effect size along with the corresponding 95% CI and SD.

My question is this: In what way can I summarize the intervention in one single, summary metric while taking into account variance for each result?

I believe this involves a random effects Meta-Analysis approach by inverse weighing of the variance, but I am not certain. I tried looking into the Metafor package in R (http://www.metafor-project.org/doku.php/analyses), but I cannot seem to find an appropriate analysis or code. Most of the examples I have seen require a sample size for multiple studies. The best I can think of is that my results are analogous to 18 different studies with a sample size of n=1, though I do not think that is a valid interpretation. As the CausalImpact methodology is based on a Bayesian approach, the CI are also not necessarily symmetrical (as seen in the example above). I am also uncertain how to present these results in an appropriate forest plot.

Any help on getting one summary metric is hugely appreciated. I apologize for any errors I may have made. Thank you.

user181973
  • 303
  • 1
  • 2
  • 8
  • Would one of the bayesain packages mentioned in the CRAN Task View help? https://CRAN.R-project.org/view=MetaAnalysis – mdewey Oct 25 '17 at 12:51
  • 1
    I am looking into them. Unfortunately, I have not had luck so far, but will update if I find the correct approach. I appreciate you bringing them to my attention. – user181973 Oct 25 '17 at 16:00
  • Is it feasible for you to present the output - Broderson et al (2015) ? And what prompted you to know standard error for your study ? and probably you do not have sample-sizes of studies ( or clinics). –  Nov 15 '17 at 04:08

2 Answers2

1

The general problem you describe is covered in the paper available here: Extending Bayesian structural time series.... They perform an analysis similar to yours but for an econometrics topic where they want to estimate the effect of a conservation program on water savings in California. To obtain the weights for the weighted meta-model in metafor they used the a tranform of the credible intervals from the causalimpact step.

Deathkill14
  • 2,140
  • 10
  • 17
0

To apply the inverse variance weighted meta-analysis method, you need a vector of effect sizes and a vector of there associated variances or squared standard errors. In your example, you have the 18 effect sizes. Is the SD that you mention the standard error of the effect size, that is, the standard error that was used to compute the confidence intervals? This would be easy to check mathematically from the computed confidence intervals. If so, the metafor will work just fine. You could use the "rma" function, giving it the effect size and variance. You can then specify the specific type of random effects model of interest, such as the method-of-moments (method="DL" -- Dersimonian and Laird). If "es" is the effect size vector and "v" is the variance vector, then the command would be "rma(es, v, method="DL")".

dbwilson
  • 1,543
  • 7
  • 10
  • Is there one such formula for Bayesian intervals to determine if it was used to compute them? My understanding is that the output provided are "credible limits", not "confidence intervals" in a frequentist sense, so I am not sure if there is a formula. Unfortunately, I am not clear as to whether the SD provided in the output is the standard error. I have looked into the documentation of the package here beginning on line 227 and cannot determine if it is so (the variable is RelEffect.sd): https://github.com/google/CausalImpact/blob/master/R/impact_inference.R – user181973 Oct 24 '17 at 19:00
  • Great question and I am not sure of the answer. It would be great to know what others think. I didn't fully think through that aspect of your question. My limited understanding of Bayesian analysis would imply that the SD is the standard deviation of the sampled posterior distribution. This is strictly speaking not a standard error but is conceptually similar, at least from a frequentist perspective. From a practical standpoint, using these SDs as SEs would produce sensible results if not "exactly" correct. That is, I'm not sure it makes sense to mix the Bayesian and Frequentists models. – dbwilson Oct 24 '17 at 19:07
  • 1
    There are Bayesian methods of meta-analysis and these accept the effect size and a measure of precision (variance) of the effect size as inputs, along with a specification of an a priori distribution for the effect size. This might be a more statistically consistent approach. – dbwilson Oct 24 '17 at 19:27
  • 1
    I appreciate the efforts and help @dbwilson. Hopefully someone can provide the exact answer. Thank you all the same. – user181973 Oct 24 '17 at 21:28