31

Probabilities of a random variable's observations are in the range $[0,1]$, whereas log probabilities transform them to the log scale. What then is the corresponding range of log probabilities, i.e. what does a probability of 0 become, and is it the minimum of the range, and what does a probability of 1 become, and is this the maximum of the log probability range? What is the intuition of this of being of any practical use compared to $[0,1]$?

I know that log probabilities allow for stable numerical computations such as summation, but besides arithmetic, how does this transformation make applications any better compared to the case where raw probabilities are used instead? a comparative example for a continuous random variable before and after logging would be good

develarist
  • 3,009
  • 8
  • 31
  • 5
    Log of probability zero is just log of zero as usual, and so indeterminate. That's not fatal to other uses. With entropy we say $p \log p \equiv 0$ whenever $p = 0$, which can be justified more rigorously. Logarithms can be useful to the extent that probabilities multiply, and for other reasons. The logarithm of a probability density can be useful too. So the logarithm of a Gaussian density is a quadratic and other distributions may helpfully be compared on log scale, as that stretches smaller densities. Log of cumulative probability or survival probability can be useful too. – Nick Cox Aug 20 '20 at 14:34
  • if entropy is defined based on log probabilities rather than probabilities, what is the advantage of transforming it further into **exponential entropy**? any significance in enveloping log probabilities with an exponential? – develarist Aug 20 '20 at 14:37
  • 7
    New questions deserve ... new questions. – Nick Cox Aug 20 '20 at 14:38
  • if it doesn't take a lot, it would still be relevant here regarding the significance of enveloping log probabilities with an exponential. since it seems the upgrades don't end with log, but seemingly exp – develarist Aug 20 '20 at 14:41
  • 5
    No; as said, comments are not for new questions. It's easy enough to find discussions of that. Also, while entropy (Shannon sense) is based on probabilities and log probabilities it is neither a probability nor a log probability. – Nick Cox Aug 20 '20 at 14:45
  • i know, and that's what got me. https://stats.stackexchange.com/questions/483969/if-entropy-has-log-probabilities-why-is-exponential-entropy-used – develarist Aug 20 '20 at 19:07
  • 1
    related: log-odds https://en.wikipedia.org/wiki/Logit – qwr Aug 21 '20 at 03:53
  • 3
    @NickCox "Indeterminate" isn't quite the right word for $\log (0)$. In this context, we can identify the symbol $\log(0)$ with the limit $\lim_{x\to0^+} \log(0) = -\infty$ in a determinate way, meaning that $\log(0) = \log(0)$ "makes sense" in a way that "indeterminate" expressions like 0/0 don't. For example, we can't say that $1/2 = \lim_{x\to0}\frac{x}{2x} = 0/0 = \lim_{x\to0}\frac{x}{x} = 1$ because $0/0$ is indeterminate. The expression $\log(0)$ does not have this ambiguity with the use of the = sign, making it determinate, despite not being a real number. – Eric Perkerson Aug 21 '20 at 22:17
  • @ericperkerson Thanks for the correction. – Nick Cox Aug 22 '20 at 00:20

4 Answers4

52

The log of $1$ is just $0$ and the limit as $x$ approaches $0$ (from the positive side) of $\log x$ is $-\infty$. So the range of values for log probabilities is $(-\infty, 0]$.

The real advantage is in the arithmetic. Log probabilities are not as easy to understand as probabilities (for most people), but every time you multiply together two probabilities (other than $1 \times 1 = 1$), you will end up with a value closer to $0$. Dealing with numbers very close to $0$ can become unstable with finite precision approximations, so working with logs makes things much more stable and in some cases quicker and easier. Why do you need any more justification than that?

develarist
  • 3,009
  • 8
  • 31
Greg Snow
  • 46,563
  • 2
  • 90
  • 159
  • is there more behind the intuition explaining how a measure that is negative and unbounded has come to be more preferable to one that is already bounded between $[0,1]$? – develarist Aug 20 '20 at 14:34
  • 3
    Log scale is not preferable and I don't think @Greg Snow is saying that either. It is just useful as explained. – Nick Cox Aug 20 '20 at 14:37
  • 14
    @develarist as the answer already mentions, if you want to represent a very, very, very small probability, then in the commonly used digital representations (e.g. IEEE 754 floating point numbers) it's preferable to store them as huge negative numbers in the logarithmic representation instead of very small positive numbers close to 0 in the direct representation, since in the latter case you'll have larger numerical errors in every calculation caused by the difference between the true value and the closest value that can be represented with the finite precision used in that encoding. – Peteris Aug 20 '20 at 23:21
  • Where the answer mentions probabilities multiplied together being closer to 0, it now has me wondering how log *joint* probabilities fare compared to their raw multiplication together. Or is that far removed from how joint probabilities (before being logged) are actually constructed? – develarist Aug 21 '20 at 13:29
  • 4
    @develarist Since `log(x*y)=log(x)+log(y)`, doing calculations of joint probablities in 'log-space' is trivial and numerically accurate. – Peteris Aug 21 '20 at 15:55
  • @Peteris Multiplication is easy but addition sucks. – kutschkem Aug 23 '20 at 08:58
  • @Peteris i would think that the digital representations of probabilities, prior to being logged, likely contain floating point errors, being the computer's fault. Doesn't this estimation error spill over into the log-transformed probabilities? – develarist Aug 24 '20 at 05:54
  • 2
    @develarist of course, some representation error is unavoidable, however, there's a quantitative difference in how much these errors accumulate over all the calculations you are going to make, since you get some extra representation error in each intermediate value, and so the scale of these (many) intermediate values and the resulting errors matter more than the scale and representation error of the initial input. Often there's a limited need to convert between log-probabilities and actual probabilities, e.g. if I need argmax or beam search, it can be done directly it in log-space. – Peteris Aug 24 '20 at 20:25
22

I would like to add that taking the log of a probability or probability density can often simplify certain computations, such as calculating the gradient of the density given some of its parameters. This is in particular when the density belongs to the exponential family, which often contain fewer special function calls after being logged than before. This makes taking the derivative by hand simpler (as product rules become simpler sum rules), and also can lead to more stable numerical derivative calculations such as finite differencing.

As an illustration, let's take the Poisson with probability function $e^{-\lambda}\frac{\lambda^{x}}{x!}$. Even though $x$ is discrete, this function is smooth with respect to $\lambda$, and becomes $\log f_x= -\lambda + x*\log(\lambda) - \log(x!)$, for a derivative with respect to $\lambda$ of simply $\frac{\partial \log f_x}{\partial \lambda} = -1 + \frac{x}{\lambda}$, which involves two simple operations. Contrast that with $\frac{\partial f_x}{\partial \lambda} = \frac{e^{-\lambda } (x-\lambda ) \lambda ^{x-1}}{x!}$, which involves natural exponentiation, real exponentiation, computation of a factorial, and, worst of all, division by a factorial. This both involves more computation time and less computation stability, even in this simple example. The result is compounded for more complex probability functions, as well as when observing an i.i.d sample of random variables, since these are added in log space while multiplied in probability space (again, complicating derivative calculation, as well as introducing more of the floating point error mentioned in the other answer).

These gradient expressions are used in both analytic and numerical computation of Maximum a Posteriori ($\ell_0$ Bayes) and Maximum Likelihood Estimators. It's also used in the numerical solution of Method of Moments estimating equations, often via Newton's method, which involves Hessian computations, or second derivatives. Here the difference between logged and unlogged complexity can be huge. And finally, it is used to show the equivalence between least squares and maximum likelihood with a Gaussian error structure.

John Madden
  • 1,752
  • 9
  • 22
7

As an example of the process mentioned in Greg Snow's answer: I quite often use high-level programming languages (Octave, Maxima[*], Gnuplot, Perl,...) to compute ratios between marginal likelihoods for Bayesian model comparison. If one tries to compute the ratio of marginal likelihoods directly, intermediate steps in the calculation (and sometimes the final result too) very frequently go beyond the capabilities of the floating-point number implementation in the interpreter/compiler, producing numbers so small that the computer can't tell them apart from zero, when all the important information is in the fact that those numbers are actually not quite zero. If, on the other hand, one works in log probabilities throughout, and takes the difference between the logarithms of the marginal likelihoods at the end, this problem is much less likely to occur.

[*] Sometimes, Maxima evades the problem by using rational-number arithmetic instead of floating-point arithmetic, but one can't necessarily rely on this.

Daniel Hatton
  • 226
  • 1
  • 3
3

This might not be what you are interested in, but log probabilities in statistical physics are closely related to the concepts of energy and entropy. For a physical system in equilibrium at temperature $T$ (in kelvin), the difference in energy between two microstates A and B is related to the logarithm of the probabilities that the system is in state A or state B:

$$E_\mathrm{A} - E_\mathrm{B} =-k_\mathrm{B}T \left[ \ln(P_\mathrm{A}) - \ln( P_\mathrm{B}) \right]$$

So, statistical physicists often work with log probabilities (or scaled versions of them), because they are physically meaningful. For example, the potential energy of a gas molecule in an atmosphere at a fixed temperature under a uniform gravitation field (a good approximation near the surface of the Earth) is $mgh$, where $m$ is the mass of the gas molecule, $g$ is the acceleration of gravity, and $h$ is the height of the molecule above the surface. The probability of finding a gas molecule in the top floor of the building versus in the bottom floor (assuming the floors have the same volume and the floor-to-ceiling height is small) is given by:

$$mg (h_\mathrm{top} - h_\mathrm{bottom}) \approx -k_\mathrm{B} T \left[ \ln (P_\mathrm{top}) - \ln(P_\mathrm{bottom}) \right]$$

This probability is trivially related to the concentration of the gas on the two floors. Higher floors have a lower concentration and the concentration of heavier molecules decays more quickly with height.

In statistical physics, it is often useful to switch back and forth between quantities proportional to log probabilities (energy, entropy, enthalpy, free energy) and quantities proportional to probability (number of microstates, partition function, density of states).

WaterMolecule
  • 81
  • 5