1

How can one measure the accuracy of the probability distribution of, say, a physical magnitude? I know one good candidate is the entropy, which measures the amount of information one has about the system---cf. Appendix A of Ref. [E. Jaynes, Physical Review 106, 620 (1957)]---. However, one can also think of the variance as a way to evaluate uncertainty: the lower it is, the better we know the system.$^*$

Imagine now that one has two probability distributions of the same variable, $P_1$ and $P_2$. Let $P_1$ have the lowest entropy and $P_2$ the lowest variance. Is it even possible to say that one provides better knowledge of the system than the other? Would another figure of merit different from the entropy and the variance answer such a question?

$^*$ As a side comment, the Heisenberg uncertainty principle is actually formulated in terms of the variance, but not the entropy of the probability distribution $|\psi(x)|^2$, $\psi(x)$ being the wavefunction.

Ferdi
  • 4,882
  • 7
  • 42
  • 62
Godoy
  • 23
  • 4

1 Answers1

1

By definition, the notion of entropy is the one that gives you the amount of knowledge in a probability distribution (from Shannon information theory).

I would say that the variance is only one part of the puzzle regarding the knowledge. Indeed, for a probability distribution defined over $\mathbb{R}$ with a given variance, you have infinite possibilities of distributions, but the one with maximum entropy is the normal distribution (with that variance).

Regarding the accuracy provided by a distribution, this is a different question. Obviously the variance is a good metric but so are the higher order moments (especially the even ones).

So it all depends on what kind of metric you are trying to get.

Here's an example: suppose your distribution is defined over $[a, b]$, with no other knowledge.

The maximum entropy distribution is $\mathcal{U}(a,b)$. The variance of this distribution is $\frac{(b-a)^2}{12}$.

Now take a beta distribution with $\alpha$ and $\beta$ close to 0. The variance tends to a higher value, $\frac{(b-a)^2}{4}$ if I'm not mistaken, but it bears a much lower entropy, because any observation is very likely to be $a$ or $b$.

Romain Reboulleau
  • 619
  • 3
  • 16
  • You distinguish between _knowledge_ and _accuracy_. So can we have more knowledge of a system with less accuracy? Does not this sound contradictory? – Godoy Oct 16 '18 at 09:03
  • Side comment: for a given entropy (e.g. _minimum_ entropy, which is the case I'm interested in), you might also have (like for the variance) infinite distributions. – Godoy Oct 16 '18 at 09:04
  • Maybe *information* is better than *knowledge*. In the case of the beta distribution with a low $\alpha$ and $\beta$ , you have much information (high entropy), because you know that the outcome is likely to be $a$ or $b$ or very close. Yet, the *accuracy* of the prediction, if you have to make any, is not better than with the uniform distribution (empirical moments are higher). – Romain Reboulleau Oct 16 '18 at 11:32
  • Agreed. In your last comment, I guess you mean '(low entropy)' instead of '(high entropy)'. Just for the sake of not confusing possible readers. – Godoy Oct 17 '18 at 10:16
  • Right, my mistake – Romain Reboulleau Oct 17 '18 at 11:45