12

Ie, to do sequential analysis (you don't know ahead of time exactly how much data you will collect) with frequentist methods requires special care; you can't just collect data until the p-value gets sufficiently small or a confidence interval becomes sufficiently short.

But when doing Bayesian analysis, is this a concern? Can we freely do things like collect data until a credible interval becomes sufficiently small?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
alecbz
  • 515
  • 3
  • 12
  • 1
    Depends. If you collect data until you have a certain amount of information that is generally not an issue, whether you are Bayesian or frequentist. If you care about frequentist operating characteristics (e.g. coverage probabilities for credible intervals, type 1 error), it is still an issue to stop e.g. once the credible interval excludes no effect. – Björn Dec 28 '15 at 18:43
  • @Björn Can you explain what "a certain amount of information" means in this context? And even if we don't get constant type 1 error rates with sequential bayesian testing, are we still "allowed" to? Can we still safely make the usual claims made in a Bayesian analysis? (ie, statements about the probability distribution of a parameter) – alecbz Dec 28 '15 at 18:53
  • 1
    Certain amount of information = e.g. Fisher information (e.g. for survival analysis up to a certain number of cases). For the second question: yes, if you use the likelihood reflecting how you sampled (i.e. e.g. reflecting, in which cases you would have stopped collecting more data). No, if you ignore what the correct likelihood is (and e.g. just use a standard normal likelihood ). – Björn Dec 28 '15 at 20:25
  • Ah, I see now, so the problem is really in the likelihood I guess. A stopping rule makes future observations conditionally dependent on prior ones. – alecbz Dec 28 '15 at 22:49
  • @Bjorn Do you know of any reference for a Bayesian analysis that takes into account a stopping rule in its liklihood function? – alecbz Dec 29 '15 at 02:19
  • Some of the references [here](http://stats.stackexchange.com/a/83970/17230) are relevant. – Scortchi - Reinstate Monica May 26 '16 at 18:45

2 Answers2

5

Rouder (2014) has a nice paper on this (written for psychologists), explaining why sequential testing (so-called data peeking) is fine from a Bayesian perspective. (Paper is freely available online if you do a search for it.)

Schoenbrodt et al. (in press) present nice analyses showing how to use sequential analysis with Bayes factors to determine when to stop data collection.

From a Bayesian parameter estimation procedure, John Kruschke has a very nice blog post that compares different Bayesian methods during sequential testing.

Hope you find them of help.

References

Rouder, Jeffrey N. (2014). Optional stopping: No problem for Bayesians. Psychonomic Bulletin & Review, 21, 301-308.

Schoenbrodt, F. D., Wagenmakers, E.-J., Zehetleitner, M., & Perugini, M. (in press). Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences. Psychological Methods.

JimGrange
  • 213
  • 1
  • 3
  • 7
0

SPRT is good example of a frequentist method that is sequential.

On the other hand, while Bayesian models have priors to overcome sparsity of data, the more data you have the "narrower" your posterior distribution becomes making it less suitable for online temporal learning.

Uri Goren
  • 1,701
  • 1
  • 10
  • 24