4

Laplace's law of succession is a well-known rule, relying on Bayes' theorem.

A possible proof of the rule of succession can be found on Wikipedia. Note that for this proof we use a uniform distribution for the parameter $p$.

Another proof of the rule is given in The Bayesian Choice as reproduced below:

Laplace succession rule

The problem is completely summarized in the image. This time, the prior we use is a uniform discrete probability distribution.

And we find both times the same final probability.

However, we did not used the uninformative prior each time. The uninformative prior for a discrete and finite set of possibilities is the uniform distribution. But the uninformative prior for the parameter $p$ in the first proof should be $1/[p(1-p)]$?

My problem is that if we use the uninformative prior in both cases (so this should only be two different formulations of the same problem ?), we find two different answers.

I am surely mistaken about the meaning of one approach, could you please give me some clue?

Xi'an
  • 90,397
  • 9
  • 157
  • 575
Unknown
  • 77
  • 7
  • 1
    You seem to be equating a uniform prior (referenced in the second line), which is a Beta$(1,1)$ distribution, with the improper Beta$(0,0)$ prior (described in the third line from the bottom). Could you please clear up this apparent contradiction? – whuber May 26 '16 at 20:23
  • Actually, I am not equating them. My problem is that if we use the Beta(0,0) prior instead of the Beta(1,1) prior in the first proof, then the two approaches give different results. This could seem normal, as we would not use a uniform prior in both cases, but to my mind it is not because I expected to find the same result if we used in both cases the uninformative prior, so that two strictly equivalent formulations of the same problem give the same result (using the uniform prior in both cases seems improper to me because then we are not considering the same problem). – Unknown May 26 '16 at 20:34
  • Interesting: it sounds like my book in French. In which case there also exists a version in English... – Xi'an May 26 '16 at 20:52
  • What exactly do you understand an "uninformative" prior to be? It's certainly the case that a Beta$(0,0)$ does not correspond to the discrete uniform distribution used in the book! – whuber May 26 '16 at 21:01
  • I call "uninformative prior" a prior that we found using the entropy maximization principle. So, to me, the Beta(0,0) corresponds in some way to the discrete uniform distribution used in the book, because they are both priors found using the principle mentioned above. It can sound weird to say that a discrete uniform distribution is transformed into a non-uniform distribution when we go from discrete to continuous, but I base myself on Jaynes' ideas (chapter 12 of Probability theory, The logic of science) - or at least of what I understood of his ideas. – Unknown May 26 '16 at 21:39
  • I found an English version of the book, so I edited the post. – Unknown May 26 '16 at 21:47
  • I would use a Beta(0,0) prior in the proof given by Wikipedia. p is then the parameter of the Bernoulli trial, that is, the probability of getting an individual of the first population (probability of success). – Unknown May 27 '16 at 07:52

1 Answers1

5

My opinion on this issue is that you are comparing the answers to two different problems, namely the Bayesian inference on the probability of "yet another sunrise" in the hypergeometric distribution and the Bayesian inference on the probability of "yet another sunrise" in the Bernoulli distribution.

There is no reason for the two answers to be equal for the same observed data.

First, given that the models are not equivalent (Bernoulli sampling cannot be turned into hypergeometric sampling), there is no principle that states that the answers should be the same. For instance, the likelihood principle does not apply there.

Second, there is no such thing as "the" non-informative or uninformative or objective prior. I discussed this in an earlier X validated answer. (Which turned out to be my most popular answer to date!) There are several coherent principles that lead to the generic construction of a reference prior, such as Jeffreys' rule, the invariance principle, the maximum entropy utility, Berger's & Bernardo's reference priors.

Third, there is a fundamental ambiguity in the definition of the maximum entropy priors in continuous settings, namely that they depend on the choice of the dominating measure. Changing the measure does change the value of the maximum entropy prior and choosing the dominating measure requires the call to yet another principle. I believe this is discussed in the Bayesian Choice to some extent.

Community
  • 1
Xi'an
  • 90,397
  • 9
  • 157
  • 575
  • 1
    So you point out that using the same principle for determining an uninformative prior in two different formulations of the same problem does not ensure to get the same results. Furthermore, in this case, there is an additional difficulty for choosing the prior, because maximum entropy priors in continuous settings depends on the domination measure used. Am I correct? – Unknown May 27 '16 at 09:43
  • 1
    You also state that Bernoulli sampling cannot be turned into hypergeometric sampling, which is obviously true when considering discrete sets. But if I realize a Bernoulli experiment, and I know that in the future I shall make $N$ draws, as there will be a fixed number of successes, the Bernoulli experiment can be turned into a discrete hypergeometric samping? And finally I do not understand what you mean by "the likelihood principle does not apply here"? – Unknown May 27 '16 at 09:54
  • The likelihood principle implies using the same prior if the likelihoods are proportional in both models but this is clearly not the case here. – Xi'an May 27 '16 at 16:51
  • OK, thank you very much for all these precise explanations. This is somehow funny because we can consider that the likelihood principle is a kind of consistency, and using maximum likelihood estimation guarantees this principle. But in the case of bayesian estimation, and despite the fact that there are some objective constraints that priors must satisfy, we lose this "consistency". So I wonder if this maximum principle could be one of the constraints when selecting a prior. I guess I now have to read your book to dig deeper. ;) – Unknown May 27 '16 at 20:17