3

We’re doing some advertising tests with test and control groups very similar to the example in the Google Research Causal Impact publication except we’re doing state tests and not DMA. I just have a couple questions as I’m reporting the analysis:

  1. Is it okay to re-run the model every several days?

    a. I noticed the historical point pred and point effect changing each iteration.

    b. Pre.period and post.period remain the same except post is extended like so: ( post.period <- c(366, nrow(disp_dat)) )

  2. Is there a minimum or maximum pre.period amount of data for training? What is best practice?

I would LOVE any help, theory explanation, advice, etc!

Here is my R Code:

pre.period <- c(1, 365) # Set the Snap Shots post.period <- c(366, nrow(data)) # So I can continually update when data is loaded in

T_C_Data <- data.frame(test,control) # (Test, Control)

CA_FL_Rev.impact <- CausalImpact(T_C_Data , pre.period, post.period, model.args = list(nseasons = 7, season.duration = 1)) # day-of-week component to data with daily granularity

Kay Brodersen
  • 1,743
  • 10
  • 10
Andre
  • 35
  • 5

1 Answers1

5

Is it okay to rerun CausalImpact() every time an additional data point has been observed?

A multiple-testing problem arises if you were to conclude that a causal effect was present as soon as one of your analyses showed a significant effect. To avoid this, you want to be more conservative in the assessment of your effects. An even better strategy is to decide a priori what post-treatment period you are interested in, and then run CausalImpact() only once.

Why do model estimates vary slightly every time the model is run?

This is because an MCMC algorithm is used for inference. This algorithm provides a stochastic approximation to the true posterior which itself is not available in closed form. You can reduce the amount of jitter between successive runs by increasing the niter argument. The default is 1000.

How long should the pre.period be?

Too short a pre-treatment period means the model may fail to pick up important patterns in your data (such as a day-of-week effect). This will result in wider posterior intervals, i.e., an underpowered analysis. A long pre-treatment period is better in principle. However, too long a pre-period means there is a chance that the structural relationship between your response variable and the predictors has changed over time. In practice, 3-10 weeks of daily data are very common. Another rule of thumb is to have about 2-3x as much data in the pre-period as in the post-period.

Kay Brodersen
  • 1,743
  • 10
  • 10