Entropy rate for continuous variables?

Question

Question: Is the concept of entropy rate defined for real-valued random processes? If so, does it have the same interpretation as in the discrete case; and how can it be estimated?

Background: As far as I know, the entropy rate of a random process is defined as:

$$ h = \lim_{n \rightarrow \infty} \frac{1}{n}H(X_1, ..., H_n) \,.$$

In conventional, finite-alphabet information theory the entropy rate of a process quantifies how many bits this process generates per realization. That is different from the entropy of the realizations themselves $H(X_i)$.

However, if the $X_i$ are continuous variables the notion of entropy $H(X_i)$ doesn't have the same interpretation as in the discrete case, since it isn't non-negative and is not invariant under change of variables.

Answer 1

You can compute the differential entropy rate of a continuous process, but like differential entropy lacks the interpretability of entropy, it will not be as meaningful as entropy rate.

To estimate the (differential) entropy rate, you can you exploit the fact that for a stationary process

$$\lim_{n \rightarrow \infty} \frac{1}{n} H[X_1, ..., X_n] = \lim_{n \rightarrow \infty} H[X_n \mid X_{n - 1}..., X_1].$$

This allows you to estimate the entropy rate by estimating the entropy of a conditional distribution. In practice, you will likely also make a Markov assumption and only take into account the last $m$ steps. Using an estimate of the conditional probability, $\hat P(x_{n} \mid x_{n - 1}, ..., x_{n - m})$, approximate the conditional entropy with the cross-entropy $$E_P[-\log \hat P(X_{n} \mid X_{n - 1}, ..., X_{n - m})].$$ This approximation is guaranteed to be an upper bound on the true entropy rate.