Study replication from a Bayesian point of view

Question

Consider following situation: Study A compares two groups and finds a mean difference with an effect size of d = .8 (p<.05). Study B is a direct replication, and finds an effect in the same direction of d = .3 (p < .05).

From an NHST perspective, there is no single/ simple answer to the question "Does Study B replicate the results of Study A?". Asendorpf et al. (2012) summarized it as follows:

"Given that no single approach to establish replicability is without limits, however, use of multiple inferential strategies […] is a better approach. In practice, this means summarizing results by answering four questions: (a) Do the studies agree about direction of effect? (b) What is the pattern of statistical significance? (c) Is the effect size from the subsequent studies within the CI of the first study? (d) Which facets of the design should be considered fixed factors, and which random factors?".

That means, there is no single answer for the question "Has this study been replicated?" (see also Sanjay Srivastava's blog post What counts as a successful or failed replication?).

Now I ask myself: Is there probably a simple(r) answer from a Bayesian point of view? Kruschke (2010) describes a "cumulative replication probability", which takes the actual posterior distribution from Study A as the prior for simulated data, which then gives an answer on the probability of replicating a certain decision of Study A (e.g. a Bayesian model comparison).

In my scenario, however, we have already two Studies conducted.

Now here's my question: Is there some way to feed Study A's posterior as information into Study B's results and come up with an answer to the question "Is Study B a replication of Study A?"? How could such an answer look like?

(As I am interested but so far unexperienced in Bayesian stats, my description of the situation might be utterly wrong ...)

---

Asendorpf. (2012). Recommendations for increasing replicability in psychology. European Journal of Personality.

Kruschke, J. K. (2010). Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science, 1(5), 658–676. doi:10.1002/wcs.72

Answer 1

Looks like you're talking about Meta-Analysis, a statistical study on previous studies.

This is not a exclusively Bayesian concept, there are many frequentist meta-analysis, but as this chapter points out, it is a good fit for Bayesian statistics.

A google search of 'bayesian meta-analysis' turns up many articles.

Answer 2

My first reaction is what's the confidence intervals on those estimates (or credible interval if it was a Bayesian analysis)?

If the confidence interval is something like,

95% CI [ .1, .5] and 95% CI [.2, .14] I'd conclude that they estimate d to be within a similar range. Even if the results of one of the studies was not significant, like, 95% CI [ -.2, .8] and 95% CI [.2, .14] I'd still conclude they found consistent results.

Answer 3

The "direct replication" you mention is an issue; whatever parameter is being assessed in Study A is also there in Study B, so it makes no sense to say an effect is e.g. positive in one and negative in the other, or zero in one and not the other, or whatever - meaning non-replication can not happen in Bayesian analyses, because of coherence.

Of course, if you are not in the situation where both studies have already been conducted (and most people planning replication are not) then it's perfectly possible to have a decision based on Study A's data be later contradicted by Study B, and this is true whether one is Bayesian or not.

Also, with different parameters in Study A and Study B, Bayesians can easily end up concluding that they differ, even if the prior information pointed elsewhere.