2

I have read that a key difference between Bayesians and Frequentists is their treatment of probability. Frequentists treat probability as the frequency with which something will happen over the long run. Bayesians treat probability as a measure of their confidence in the outcome of a single event.

I've also read that Frequentists consider models to be fixed, and data to vary; while Bayesians consider models to vary and data to be fixed.

How exactly do the different treatments of probability lead to these different views of data/models?

And then, the crux of my question is, why do these different views allow Bayesians to talk about the probability of a hypothesis being true given some data, while Frequentists are restricted to talking about the probability of data being true given some hypothesis?

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
Dave
  • 1,641
  • 2
  • 14
  • 27
  • 1
    It's a very trivialized characterization of approaches. You use statistics to make decisions. Suppose, you're analyzing the production process to identify the cause of the deviations from the quality threshold. Whether you're frequentist or Bayesian in the end you need to make a decision whether to change the parameter of the process or not. – Aksakal Mar 07 '18 at 20:07
  • I recently asked a very similar [question](https://stats.stackexchange.com/questions/332026/can-we-think-of-a-probability-in-both-the-classical-and-subjective-sense-simulta) that may be of interest. – Kareem Carr Mar 07 '18 at 21:24

1 Answers1

4

To perhaps overly simplify, a key distinction is that Bayesian statisticians are willing to use tools of probability in areas the frequentist wouldn't. Imagine we have a thumbtack that could land heads or tails. Let the probability of landing heads be $\theta$.

  • In frequentist statistics, $\theta$ is a real number.
  • In Bayesian statistics, $\theta$ is a random variable.

Further explanation.

  • To the frequentist, the probability $\theta$ a thumbtack will land heads is a real number. We don't know what that number is $\theta$, but the parameter is a single real number in $[0, 1]$. Talking about the probability that $\theta > .5$ is bizarre because $\theta$ is not a random variable! It's a number. Asking for $P(\theta > .5)$ is like asking $P(1>2)$ or $P(2<3)$. $\theta > .5$ has no uncertainty.

  • In contrast, the subjective Bayesian is willing to model her uncertainty (in her own head) over $\theta$'s value using the tools of probability. To the subjective Bayesian, the probability $\theta$ a thumbtack will land heads is a random variable: it's a measurable function from some sample space to the space of real numbers. The expression $\theta > .5$ is another random variable: a measurable function from the sample space $\Omega$ to the space of binary values $\{0, 1\}$.

If you have access, David Kreps's Notes on the Theory of Choice has a beautiful dialogue that brings out these concepts (and from which I'm drawing the thumbtack example).

Once you treat $\theta$ as a random variable, you can talk about $P(\theta \mid X)$, the probability of parameter values given the data (using Bayes rule). It also raises the question of what's the unconditional distribution of $\theta$ (i.e. the prior).

Matthew Gunn
  • 20,541
  • 1
  • 47
  • 85