3

I'm looking for an intuition, apologies for the non-technical language.

In Bayesian Learning one starts from prior knowledge and incorporates a new fact to obtain posterior knowledge.

What is the weight of the prior knowledge VS the weight of the new fact, in determining the posterior knowledge?

  • On one extreme, I could totally change my mind at any new fact. if my prior knowledge says 1, and the fact says 2, then my posterior knowledge will say 2

  • On the other extreme, I could be stubborn and hardly change my mind in face of new facts. If my prior knowledge says 1 and the new fact says 2, my posterior knowledge will say 1.001

Is there a parameter that regulates how quickly I change my mind in face of new facts?

elemolotiv
  • 1,048
  • 7
  • 20
  • 3
    I don't use Bayesian estimation, but I believe that the answer is connected the prior's variance. For example, compare N(1, 100) to N(1, 2) to N(1, 1). I believe the first would count as a vague. The latter two have much smaller variance. The smaller the prior's variance, the more the prior will impact the likelihood (albeit I don't know how much). N(1, 0) would indicate complete certainty, so the data don't impact the likelihood at all - albeit I don't think anyone should do that. – Weiwen Ng Oct 28 '19 at 20:59
  • 1
    @WeiwenNg In general, it's not the variance. Consider the counterexample of a prior consisting of two delta functions spaced arbitrarily far apart. The variance can be made arbitrarily large, but the prior will completely overwhelm the likelihood. – user20160 Oct 28 '19 at 21:56
  • @user20160 if not the variance, then what is in general that regulates the learning rate? – elemolotiv Oct 29 '19 at 05:24
  • 1
    Please define what exactly do you mean by "stubbornness". Your question is unclear, what makes it hard, to impossible to answer. Please edit it, to make it more precise, after that we can re-open it and you can put the bounty again to it. (Your current bounty was refunded when the question got closed.) – Tim Nov 04 '19 at 19:59
  • Thanks for clarifying. I'm re-opening it. If you still wish, you can put the bounty back. – Tim Nov 07 '19 at 09:25
  • Thanks Tim, let's do it re-open – elemolotiv Nov 07 '19 at 09:27

2 Answers2

5

In non-pathological examples, the prior's variance plays an important role in the "stubbornness" of estimates, especially in hierarchical models. This is very well shown in the 8 schools examples in BDA3.

Here is a photo which speaks to this. On the horizontal axis is the prior's standard deviation. As the standard deviation shrinks (alternatively, precision increases) the estimates are pooled closer to the population mean estimate. As the standard deviation increases (precision decreases) the the pooling effect is relaxed, and estimates head towards their completely pooled values.

enter image description here

Demetri Pananos
  • 24,380
  • 1
  • 36
  • 94
1

Bayes theorem is

$$ p(\theta | X) \propto {\overbrace{\vphantom{\prod_{i=1}^N} p(\theta)}^\text{prior}} \; {\overbrace{\prod_{i=1}^N p(X_i | \theta)}^\text{likelihood}} $$

so as the sample size $N$ grows large, the likelihood would play greater role in the posterior. Informally, likelihood enters the formula $N$ times, while the prior only once. Of course this is not that simple.

As you already noticed, you could use a degenerate prior $p(\theta = c) = 1$ and $p(\theta \ne c) = 0$, that would be zero for any value other then $c$ (because of multiplying by zero). In such case, the prior would zero-out everything that is inconsistent with it. Another extreme would be "uninformative" prior $p(\theta) \propto 1$ that plays no role in the posterior (because you always multiply by same constant). Everything in-between depends on how peaked is your prior around the value of interest, where the higher values you assign to some value, the more influence would the prior have on the outcome.

The likelihood would usually de determined by the data, in the sense that you don't manipulate it to influence the results, it is only used to describe the model in probabilistic terms. As you can see, all the data points (the new "facts") have same "weight".

So, you regulate how much the prior influences the posterior by making the prior more, or less, informative. There is no additional parameter for this.

Tim
  • 108,699
  • 20
  • 212
  • 390