Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [maximum-entropy]

maximum entropy or maxent is a statistical principle derived from information theory. Distributions maximizing entropy (under some constraints) are thought to be "maximally uninformative" given the constraints. Maximum entropy can be used for multiple purposes, like choice of prior, choice of sampling model, or design of experiments.

164 questions
41
votes
8 answers

Why is Entropy maximised when the probability distribution is uniform?

I know that entropy is the measure of randomness of a process/variable and it can be defined as follows. for a random variable $X \in$ set $A$ :- $H(X)= \sum_{x_i \in A} -p(x_i) \log (p(x_i)) $. In the book on Entropy and Information Theory by…
user76170
  • 639
  • 2
  • 8
  • 9
31
votes
3 answers

Entropy-based refutation of Shalizi's Bayesian backward arrow of time paradox?

In this paper, the talented researcher Cosma Shalizi argues that to fully accept a subjective Bayesian view, one must also accept an unphysical result that the arrow of time (given by the flow of entropy) should actually go backwards. This is mainly…
ely
  • 2,272
  • 18
  • 31
30
votes
4 answers

Statistical interpretation of Maximum Entropy Distribution

I have used the principle of maximum entropy to justify the use of several distributions in various settings; however, I have yet to be able to formulate a statistical, as opposed to information-theoretic, interpretation of maximum entropy. In other…
Annika
  • 301
  • 3
  • 3
24
votes
4 answers

Weakly informative prior distributions for scale parameters

I have been using log normal distributions as prior distributions for scale parameters (for normal distributions, t distributions etc.) when I have a rough idea about what the scale should be, but want to err on the side of saying I don't know much…
John Salvatier
  • 4,032
  • 1
  • 18
  • 28
18
votes
2 answers

Prove that the maximum entropy distribution with a fixed covariance matrix is a Gaussian

I'm trying to get my head around the following proof that the Gaussian has maximum entropy. How does the starred step make sense? A specific covariance only fixes the second moment. What happens to the third, fourth, fifth moments etc?
Tarrare
  • 195
  • 1
  • 8
16
votes
2 answers

What is the maximum entropy probability density function for a positive continuous variable of given mean and standard deviation?

What is the maximum entropy distribution for a positive continuous variable, given its first and second moments? For example, a Gaussian distribution is the maximum entropy distribution for an unbounded variable, given its mean and standard…
becko
  • 3,298
  • 1
  • 19
  • 36
15
votes
1 answer

Is differential entropy always less than infinity?

For an arbitrary continuous random variable, say $X$, is its differential entropy always less than $\infty$? (It's ok if it's $-\infty$.) If not, what's the necessary and sufficient condition for it to be less than $\infty$?
syeh_106
  • 746
  • 4
  • 15
13
votes
1 answer

Maximum Entropy with bounded constraints

Assume we have the problem of estimating the probabilities $\{p_1,p_2,p_3\}$ subject to: $$0 \le p_1 \le .5$$ $$0.2 \le p_2 \le .6$$ $$0.3 \le p_3 \le .4$$ with only the natural constraint of $p_1+p_2+p_3=1$ Found two compelling arguments using…
sheppa28
  • 1,287
  • 8
  • 15
13
votes
1 answer

Are there any contemporary uses of jackknifing?

The question: Bootstrapping is superior to jackknifing; however, I am wondering if there are instances where jackknifing is the only or at least a viable option for characterizing uncertainty from parameter estimates. Also, in practical situations…
N Brouwer
  • 1,786
  • 2
  • 18
  • 32
12
votes
1 answer

What distribution has the maximum entropy for a known mean absolute deviation?

I was reading the discussion on Hacker News about the use of the standard deviation as opposed to other metrics such as the mean absolute deviation. So, if we were to follow the principle of maximum entropy, what kind of distribution would we use…
Dietrich Epp
  • 223
  • 1
  • 6
12
votes
2 answers

How to determine Forecastability of time series?

One of the important issues being faced by forecasters is if the given series can be forecasted or not ? I stumbled on an article entitled "Entropy as an A Priori Indicator of Forecastability" by Peter Catt that uses Approximate Entropy (ApEn) as…
forecaster
  • 7,349
  • 9
  • 43
  • 81
12
votes
3 answers

Maximum likelihood estimator of joint distribution given only marginal counts

Let $p_{x,y}$ be a joint distribution of two categorical variables $X,Y$, with $x,y\in\{1,\ldots,K\}$. Say $n$ samples were drawn from this distribution, but we are only given the marginal counts, namely for $j=1,\ldots,K$: $$ S_j =…
R S
  • 507
  • 1
  • 5
  • 15
11
votes
2 answers

When does the maximum likelihood correspond to a reference prior?

I have been reading James V. Stone's very nice books "Bayes' Rule" and "Information Theory". I want to know which sections of the books I did not understand and thus need to re-read further. The following notes which I wrote down seem…
Chill2Macht
  • 5,639
  • 4
  • 25
  • 51
10
votes
1 answer

Is the maximum entropy distribution consistent with given marginal distributions the product distribution of the marginals?

There are generally many joint distributions $P(X_1 = x_1, X_2 = x_2, ..., X_n = x_n)$ consistent with a known set marginal distributions $f_i(x_i) = P(X_i = x_i)$. Of these joint distributions, is the product formed by taking the product of the…
wnoise
  • 515
  • 4
  • 16
9
votes
1 answer

Bayesian vs Maximum entropy

Suppose that the quantity which we want to infer is a probability distribution. All we know is that the distribution comes from a set $E$ determined, say, by some of its moments and we have a prior $Q$. The maximum entropy principle(MEP) says that…
Ashok
  • 1,011
  • 7
  • 22
1
2 3
…
10 11