19

Frank Harrell has started a blog (Statistical Thinking). In his premier post, he lists some key features of his statistical philosophy. Among other items, it includes:

  • Make the sample size a random variable when possible
  1. What does it mean to "make the sample size a random variable"?
  2. What are the advantages of doing this? Why might it be preferable?
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
  • In sequential analysis the time of occurrence of an event is treated as a random variable. That is also true abot sample size. – Michael R. Chernick Jan 17 '17 at 03:57
  • @RichardHardy, this should be discussed on [meta.stats.SE]. I created the tag b/c we didn't have 1 & there are a lot of questions about ACF, etc. We could always make it a synonym. – gung - Reinstate Monica May 01 '18 at 10:50

1 Answers1

14

I'm not meaning to use models close to the data collecting process but rather doing continuous Bayesian monitoring of posterior probabilities, which require no penalty for multiplicity. Instead of computing an arbitrary target sample size I'd prefer to compute a maximum possible sample size (for budget approval) and otherwise to stop "when we get the answer" as usually done to good effect in physics. I'll say more about that in my blog http://fharrell.com some day before long.

Frank Harrell
  • 74,029
  • 5
  • 148
  • 322
  • 1
    What does "when we get the answer" mean concretely? I would think that running a study until you got a result you liked (eg, a 95% credible interval doesn't include 0) would be just as corrupting in a Bayesian context as in a frequentist one. – gung - Reinstate Monica Jan 17 '17 at 21:30
  • 1
    @gung not at all. Bayesian inference is completely independent of the stopping rule. It's easy to simulate the calibration of posterior probabilities at the time of early stopping, showing they are exactly correct. This is one of the amazing differences with the frequentist world. In general, forward probabilities are context-free and backward probabilities depend on how you got there. So I would stop when the posterior probability of the effect being > 0 exceeds some number such as 0.95 or when the credible interval has width < some specified number. – Frank Harrell Jan 17 '17 at 23:38
  • 1
    Your response to @gung's comment seems to me to beg the question: some readers may well feel that if Bayesian inference indeed allows "sampling to a foregone conclusion", so much the worse for Bayesian inference. (I'd refer them to the references in the 3rd paragraph [here](http://stats.stackexchange.com/a/83970/17230).) Looking forward to your next blog post! – Scortchi - Reinstate Monica Jan 18 '17 at 09:59
  • Sampling to a foregone incorrect conclusion only happens if the prior used by statistician conflicts with the prior used by the reviewer. For example if the reviewer puts a probability mass at the null (i.e., the prior has an absorbing state) and the model used does not put special emphasis on the null, the analysis may indicate stopping for a positive effect but the reviewer says there is insufficient evidence for an effect. If you simulate studies with a certain prior and analyze using the same prior, the posterior probs are perfectly calibrated, and the posterior means are perfect also. – Frank Harrell Jan 18 '17 at 17:10