1

As far as I know, when we have just data and no constraints (other than probabilities must add up to 1), the distribution that gives maximum entropy is uniform distribution. But when we know mean and variance, we add 2 more constraints so the distribution that gives maximum entropy is Gaussian.

But how can we decide that? Doesn't all the data we collect have a mean and variance, so doesn't it all cases it is normal distribution? And what about other distributions? I don't know how we can decide, for example, chi-square distribution gives maximum entropy etc, in which circumstances?

I think deciding what constraints we have confuses me, and for me, with my current limited knowledge probably, I can't see a case where I can choose uniform over normal distribution since I can calculate the mean and variance of my data.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
ikadorus
  • 21
  • 1

1 Answers1

1

Sorry, but you got it wrong.

  • Without any constraints there is no maxent distribution, as for instance uniform distributions on $[-n,n]$ have entropies going to infinity with $n$. You must have some constraints.

  • Typically, the constraints do not come from the data, they come from some external considerations. If you are interested in distributions on the interval $[a,b]$, the limits $a,b$ do not come from the data, they come from some external "prior" knowledge of the system.

  • While you can always compute empirical means/variances from finite empirical data, they do not always have meaning. See for instance. See for instance What is the difference between finite and infinite variance. The mean/variave constraints used in maxent typically are theoretical, model parameters. And not all theoretical distributions have means or variances. See for instance Test for finite variance?.

  • As for choosing between finite range and mean/variance description, that do not come from the data, it must come from prior information about the system!

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467