Is there more to probability than Bayesianism?

Question

As a student in physics, I have experienced the "Why I am a Bayesian" lecture perhaps half a dozen times. It is always the same -- the presenter smugly explains how the Bayesian interpretation is superior to the frequentist interpretation allegedly employed by the masses. They mention Bayes rule, marginalization, priors and posteriors.

What is the real story?

Is there a legitimate domain of applicability for frequentist statistics? (Surely in sampling or rolling a die many times it must apply?)

Are there useful probabilistic philosophies beyond "bayesian" and "frequentist"?

Answer 1

The Bayesian interpretation of probability suffices for practical purposes. But even given a Bayesian interpretation of probability, there is more to statistics than probability, because the foundation of statistics is decision theory and decision theory requires not only a class of probability models but also the specification of a optimality criteria for a decision rule. Under Bayes criteria, the optimal decision rules can be obtained through Bayes' rule; but many frequentist methods are justified under minimax and other decision criteria.

Answer 2

Take a look at this paper by Cosma Shalizi and Andrew Gelman about philosophy and Bayesianism. Gelman is a proeminent bayesian and Shalizi a frequentist!

Take a look also at this short criticism by Shalizi, where he points the necessity of model checking and mock the dutch book argument used by some Bayesians.

And last, but not least, I think that, since you are a physicist, you may like this text, where the author points to “computational learning theory” (which I frankly know nothing at all), which could be an alternative to Bayesianism, as far as I can understand it (not much).

ps.: If you follow the links, specially the last one and have an opinion about the text (and the discussions that followed the text at the blog of the author)

ps.2: My own take on this: Forget about the issue of objective vs subjective probability, the likelihood principle and the argument about the necessity of being coherent. Bayesian methods are good when they allow you to model your problem well (for instance, using a prior to induce unimodal posterior when there is a bimodal likelihood etc.) and the same is true for frequentist methods. Also, forget about the stuff about the problems with p-value. I mean, p-value sucks, but in the end they are a measure of uncertainty, in the spirit of how Fisher thought of it.

Answer 3

"Bayesian" and "frequentist" aren't "probabilistic philosophies". They're schools of statistical thought and practice concerned mainly with quantifying uncertainty and making decisions, although they're often associated with particular interpretations of probability. Probably the most common perception, although it is incomplete, is that of probability as subjective quantification of belief versus probabilities as long-run frequencies. But even these aren't really mutually exclusive. And you may not be aware of this but there are avowed Bayesians who don't agree on particular philosophical issues about probability.

Bayesian statistics and frequentist statistics aren't orthogonal either. It seems like "frequentist" has come to mean "not Bayesian" but that's incorrect. For example, it's perfectly reasonable to ask questions about the properties of Bayesian estimators and confidence intervals under repeated sampling. It's a false dichotomy perpetuated at least in part by a lack of a common definition of the terms Bayesian and frequentist (we statisticians have no one to blame but ourselves for that).

For an amusing, pointed and thoughtful discussion I would suggest Gelman's "Objections to Bayesian Statistics", the comments, and the rejoinder, available here:

http://ba.stat.cmu.edu/vol03is03.php

There is even some discussion about confidence intervals in physics IIRC. For more in-depth discussions you could walk back through the references therein. If you want to understand the principles behind Bayesian inference, I would suggest Bernando & Smith's book but there are many, many other good references.

Answer 4

There are non-Bayesian systems or philosophies of probability -- Baconian & Pascalian, e.g. If you are into epistemology & philosophy of science you might enjoy the debates--otherwise, you'll shake your head & conclude that in fact the Bayesian interpretation is all there is.

For good discussions,

• Cohen, L.J. An introduction to the philosophy of induction and probability, (Clarendon Press ; Oxford University Press, Oxford New York, 1989)
• Schum, D.A. The evidential foundations of probabilistic reasoning, (Wiley, New York, 1994).

Answer 5

For me, the important thing about Bayesianism is that it regards probability as having the same meaning we apply intuitively in everyday life, namely the degree of plausibility of the truth of a proposition. Very few of us really use probability to mean strictly a long run frequency in everyday use, if only because we are often interested in particular events that have no long run frequency, for example what is the probability that fossil fuel emissions are causing significant climate change? For this reason, Bayesian statistics are much less prone to misinterpretation than frequentist statistics.

Bayesianism also has marginalisation, priors, maxent, transformation groups etc. that all have their uses, but for me the key benefit is that the definition of probability is more appropriate for the kinds of problems I want to address.

That doesn't make Bayesian statistcs better than frequentist statistics. It seems to me that frequentist statistics are well suited to problems in quality control (where you do have repeated sampling from populations) or where you have designed experiments, rather than analysis of pre-collected data (although that lies rather beyond my expertise, so it is just intuition).

As an engineer, it is a matter of "horses for courses" and I have both sets of tools in my toolbox and I use both on a regular basis.