8

In the Bayesian theory of probability, probability is our expression of knowledge about a certain thing, not a property of that thing. However, I always see people treat $p$ as a parameter that needs to be estimated. They set up a prior for $p$, usually in the form of a beta function and then update it as "realizations" of this variable come in.

Even the great bayesian Jaynes sometimes gives the impression that he is "estimating the probabilities" or looking for the $p$ which best "fitst the data":

Now we wish to take into account only the hypotheses belonging to the ‘Bernoulli class’ $B_m$ in which there are $m$ possible results at each trial and the probabilities of the $A_k$ on successive repetitions of the experiment are considered independent and stationary;

Probability Theory, E. T. Jaynes, page 297

This makes me confused, because $p$ is not a probability, since it is a property of the random variable and it is not a frequency, since the variable represents a single event.

Martin Drozdik
  • 613
  • 4
  • 14
  • 1
    In the cited section, is Jaynes in fact providing a Bayesian treatment of the interpretation and design for the estimation of $p$? Even a great Bayesian can conceivably write a book called "Probability Theory" that deals with both frequentist and Bayesian interpretation of probability. – AdamO Mar 08 '19 at 16:20
  • 1
    If we are to be strict epistemiologists, $p$ is a *belief* for the Bayesian: something which is not fixed but has uncertainty which can be described with a probability distribution. If we need to update our belief, we perform an experiment. In isolation, the experiment produces a likelihood and, consequently, an MLE which has all the frequentist interpretations, but when it updates the prior, attributes of the posterior can be quite different. The Posterior mode may not be the MLE, the posterior median may not be the median unbiased estimate. – AdamO Mar 08 '19 at 16:28
  • Just a caveat, the frequentist interpretation of the experiment is acceptable to the Bayesian iff they believe infinite, independent replicatability is approximately doable. – AdamO Mar 08 '19 at 16:31
  • Are you interested specifically in the Bayesian perspective or just in general? Either way, $p$ is a parameter, nothing more. What does it matter if you call it a probability or something else? As you said, it is not a PDF, only a realisation or the output of a PDF if you will. – Digio Mar 08 '19 at 16:38
  • 1
    @Digio I am interested only in the Bayesian perspective. I am trying to imagine how I would coherently explain what a Bernoulli random variable really is to a stranger. What words would I use. The more I think about the problem, the more am I inclined to call it a probability. On the other hand, I would avoid both probability and frequency when talking about measurable things. I would maybe say that we are estimating the average proportion of events that have a given property. – Martin Drozdik Mar 08 '19 at 16:55
  • 1
    I think you're mixing up different concepts here. The nature of parameter $p$ (a realisation of a discrete uniform r.v.) should be independent of the inference framework, which will view that parameter as fixed or random variable. Then, what is a Bernoulli r.v. is again a different concept, not too difficult to explain IMHO, but certainly not directly related to the nature of $p$. – Digio Mar 08 '19 at 21:03

3 Answers3

3

$p$ is a parameter which specifies the "success probability", for which we have prior and posterior probability distributions.

We may for example have a coin for which we are not sure if it is fair ($p=0.5$) or not ($p\neq 0.5$). Even so, fairness, or lack thereof, is a property of the coin. We just happen to be unsure about that property of the coin.

We then, for example, specify a beta prior distribution as a prior probability distribution over the possible success probabilites in $[0,1]$. That prior may, for example, be inspired by looking at the coin, assessing if it "looks" fair. If it looks fair, we will be inclined to specify a prior with lots of probability mass around $p=0.5$.

In other cases, say, when forming a prior belief about the probability with which a football player will be successful at his next penalty - also a Bernoulli outcome, either a goal or not - we will be inclined to put more probability mass on $p$ around 0.8, because professional football players score on most penalties.

We then toss the coin/observe the player a couple of times, and summarize the information in the likelihood function, to obtain the update, i.e., the posterior.

Christoph Hanck
  • 25,948
  • 3
  • 57
  • 106
  • Thank you for your kind answer. However, in the first sentence you say that p is a probability. This, as far as I know, is in conflict with the rest of your answer, where you treat p as a property of the physical world (speaking about knowledge about p, priors for p, ...). If I understand bayesian probability theory correctly, there is no concept of "probability of probability". – Martin Drozdik Mar 08 '19 at 14:07
  • I am sorry for not explaining my view convincingly, but I do not see the issue. It is a probability which may be interpreted in the "frequentist" sense - if you toss the coin an infinite number of times, it will show heads $p*100\%$ of the time when $p$ is the success probability. – Christoph Hanck Mar 08 '19 at 14:11
  • Especially in the second paragraph you mention that fairness is a property of the coin. I would disagree. Maybe the location of the center of mass is a property of the coin, but probability is in your mind. You cannot be unsure whether p=0.5 or not. In this paradigm of reasoning, you simply have a p. – Martin Drozdik Mar 08 '19 at 14:11
  • The center of the mass will affect how often heads will show up, so that physical feature of the coin will affect the parameter of interest. – Christoph Hanck Mar 08 '19 at 14:12
  • Would it maybe change your view if we wrote the parameter as an expected value w.r.t. an indicator function, $p=P(Y=1)=E(I_{\{Y=1\}})$, more akin to what we have, for example, in regression models, where $E(y|X)=X\beta$? – Christoph Hanck Mar 08 '19 at 14:18
  • But if the probability were really a physical property of the coin, then how would we update it after seeing heads 100 times in a row? Wouldn't this be akin to somehow "updating" the physical properties of the real world, such as updating the mass of a hydrogen atom? – Martin Drozdik Mar 08 '19 at 14:18
  • I am not trying to be facetious, I am grateful for your explanation, but I just don't get it. I will try to spend more time thinking. – Martin Drozdik Mar 08 '19 at 14:20
  • Indeed, what you update is *your assessment of the fairness of the coin*, which you summarize in a probability distribution. To try and separate notation: Let $\pi$ be the distribution summarizing your assessment, and $p$ the property of the coin, which you are unsure about. What you update is $\pi$, not $p$, from the prior $\pi(p)$ to the posterior $\pi(p|y)$. – Christoph Hanck Mar 08 '19 at 14:21
  • 1
    I think there's a little trick being done here. While Bayesians have a certain definition of probability, they also know that frequentists exist. Bayesians can acknowledge that a coin has a property, $p$, which can be stated as follows: "the coin's property $p$ is what a frequentist would measure as the 'probability' of it landing on heads if they had infinite trials with the coin". – Bridgeburners Mar 08 '19 at 14:23
3

For a random variable $X \sim \operatorname{Bernoulli}(p)$ defined on a probability space $(\Omega, \mathcal{F}, P)$, the parameter $p$ (a number) is the probability of a certain event, namely, the event $\{X = 1\}$. That is, $$ p = P(X = 1). $$ The single number $p$ completely determines the distribution of $X$ since for any Borel set $B \subseteq \mathbb{R}$ we have $$ \begin{aligned} P(X \in B) &= \mathbf{1}_B(0)P(X = 0) + \mathbf{1}_B(1) P(X = 1) \\ &= (1 - p) \mathbf{1}_B(0) + p \mathbf{1}_B(1). \end{aligned} $$ (Here $\mathbf{1}_B$ is the indicator function of $B$.) This is why the family of Bernoulli distributions is parametereized by the interval $[0, 1]$. This fact is independent of a frequentist or Bayesian interpretation of statistics: it is just a fact of probability.

If we're being Bayesians, then we'd want the parameter $p$ to be a random variable itself with some prior distribution. Formally, we can say that our parameter is a random variable $\Pi$ supported on $[0, 1]$ and we have $$ X \mid \Pi \sim \operatorname{Bernoulli}(\Pi), $$ which means that $$ \begin{aligned} P(X = 1 \mid \Pi) &= \Pi, & P(X = 0 \mid \Pi) &= 1 - \Pi \end{aligned} $$ almost surely (or $$ \begin{aligned} P(X = 1 \mid \Pi = p) &= p, & P(X = 0 \mid \Pi = p) &= 1 - p \end{aligned} $$ for any $p \in [0, 1]$). In this case, the parameter $\Pi$ (a random variable) is the conditional probability of the event $\{X = 1\}$ given $\Pi$. This conditional probability, together with the distribution of $\Pi$ (the prior distribution), completely determines the distribution of $X$ since$$ \begin{aligned} P(X \in B) &= E[P(X \in B \mid \Pi)] \\ &= E[\mathbf{1}_B(0)P(X = 0 \mid \Pi) + \mathbf{1}_B(1) P(X = 1 \mid \Pi)] \\ &= E[(1 - \Pi) \mathbf{1}_B(0) + \Pi \mathbf{1}_B(1)] \\ &= (1 - E[\Pi]) \mathbf{1}_B(0) + E[\Pi] \mathbf{1}_B(1) \end{aligned} $$ for any Borel set $B \subseteq \mathbb{R}$.

In any case, frequentist or Bayesian, the usual parameter of Bernoulli data is the probability (either marginal or conditional) of some event.

Artem Mavrin
  • 3,489
  • 1
  • 16
  • 26
3

In the Bayesian theory of probability, probability is our expression of knowledge about a certain thing, not a property of that thing. However, I always see people treat $p$ as a parameter that needs to be estimated. They set up a prior for $p$, usually in the form of a beta function and then update it as "realizations" of this variable come in.

This is irrelevant. It has nothing to do with interpreting the meaning of probability, since this is not about philosophy, but about well defined mathematical object. You see people discussing estimating value of $p$ because you look into statistics handbooks and statistics is about estimating things, but $p$ is a parameter of distribution, it can be known, or unknown.

If $X$ is a Bernoulli random variable with probability of "success" $p$, then $\Pr(X=1) = p$ by definition. So $p$ is a parameter of this distribution, but it is also probability of "success".

This makes me confused, because $p$ is not a probability, since it is a property of the random variable and it is not a frequency, since the variable represents a single event.

Yes, random variable describes some "single event", so if you are going to toss a coin, the possible outcome is a random variable because it is uncertain. After you tossed the coin and know the outcome, it is no more random, the outcome is certain. As about probability, in frequentist setting you consider hypothetical scenario where you would repeat the coin tossing experiment huge number of times and the probability would be equal to the proportion of heads among those repetitions. In subjective, Bayesian setting, the probability is a measure of how much do you believe that you will observe heads.

The above is however irrelevant to question what $p$ is. It is a parameter that is also equal to probability of "success". The question how do you interpret the probability and what does it mean is a different question.

Tim
  • 108,699
  • 20
  • 212
  • 390