7

I'm a frequentist but have been exploring Bayesian techniques lately. One thing that bothers me is that I'm never completely sure that I've implemented them correctly. To be specific, I'm not talking about convergence, burn-in time, or autocorrelation. I'm talking about, 'did I code the formulas correctly?'.

What I do now is to jack up the sample size and watch for the posterior to hug the true parameter values (with simulated data). But that's completely ignoring the power of the prior and the great small sample properties that Bayesians love to talk about.

Frequentist techniques make testable predictions in the form of relative frequencies. But in the Bayesian context, we lose the relative frequency-based interpretation of probability. So how does one programmatically test a Bayesian method's software implementation?

Ben Ogorek
  • 4,629
  • 1
  • 21
  • 41

1 Answers1

8

Bayesians don't lose the relative frequency-based interpretation of probability. In particular, if you define this procedure:

  1. simulate from the prior,
  2. then simulate from the model using those values from the prior, and
  3. estimate the parameters using the same prior.

Then your credible intervals should have the appropriate frequentist coverage, i.e. 95% intervals should include the true parameter in 95% of your analyses, over repeated replicates of the procedure.

jaradniemi
  • 4,451
  • 13
  • 25
  • Okay, so you if you treat the prior as part of the data generating process, and the posterior makes testable frequency-based predictions. I'll keep this question open for a few more days, but I like this answer. – Ben Ogorek May 07 '14 at 05:07
  • For further information on this procedure see the references at https://stats.stackexchange.com/questions/513100/official-name-of-a-common-type-of-bayesian-simulation-study – S. Catterall Sep 04 '21 at 22:28