2

As you know, the likelihood of a multivariate Gaussian mixture model with C components and $d$ dimensional $x$ is as follow: $$p(x|\lambda) = \Sigma^{C}_{c=1}w_c p(x_t|\mu_c,\Sigma_c)$$ where $\lambda=\{w,\mu, \Sigma\}$, $w_c$ is the weight for component $c$ and $p(x_t|\mu_c,\Sigma_c)$ is Gaussian probability density function with mean $\mu_c$ and covariance matrix $\Sigma_c$. consequently, the log-likelihood would be: $$log(p(x|\lambda))=log(\Sigma^{C}_{c=1}w_c p(x_t|\mu_c,\Sigma_c))$$

now my question: is there any way to get rid of the log before the sum? It causes some numerical instabilities because of the exponential in the normal PDF, when the covariance matrix becomes very small. If log taken inside the sum, then:$$log(exp(X)) = X$$

I am looking for a solution for the general case for a $d$ dimensional $x$ and $C$ components GMM, with the PDF as follow:

PDF of a multivar. normal dist.

Any ideas?

PickleRick
  • 688
  • 6
  • 19
  • 1
    Why is it causing instabilities? The probabilities are always strictly positive, because the pdf of the normal, as you mentioned, is an exponential, so even if some of the weights are zero, the sum will be strictly positive, and so the logarithm will be well-defined. Also, the logarithm is increasing and preserves concavity, so it is well behaved respect to the maximixation. – Anna SdTC Mar 14 '17 at 09:01
  • 1
    A possible solution is to look for the largest term in the sum for each $x_t$ say $w_1 p(x_t|\mu_1,\Sigma_1)$, and to get this term factor the sum,$$w_1 p(x_t|\mu_1,\Sigma_1)\sum_{c=1}^Cw_c p(x_t|\mu_c,\Sigma_c)\big/w_1 p(x_t|\mu_1,\Sigma_1)$$If this is not enough, you may have to find the maximum of the maxima over all $x_t$ and factor again by this term... – Xi'an Mar 14 '17 at 09:22
  • @AnnaSdTC I have a limit to store numbers in float32. The upper limit is exp(80). So if my covariance has any value smaller than 1/80, I will get inf. – PickleRick Mar 14 '17 at 09:28
  • @Xi'an this is a good suggestion, with one problem: $p(x_t|\mu_c,\Sigma_c)$ is already unstable since the exp is in there. scaling after getting inf would not help. – PickleRick Mar 14 '17 at 09:34
  • @Xi'an $log (\Sigma^{C}_{c=1} f(x)) != \Sigma^{C}_{c=1} log( f(x))$. Or I perhaps I misunderstood what you were suggesting. Did I get your suggestion right? let me try to explain myself better: for a component $c$, before taking the log, the overflow might be happened already. So taking the logarithm for a component wouldn't help if the overflow already happened. – PickleRick Mar 14 '17 at 09:43
  • 1
    @Xi'an Usually in here, the trick is to come up with an extended version of the log-likelihood. There are cases that this is already available for the 2 dimensional case, where the formula is extended and the log is inside the sum over the components and consequently, log(exp()) will solve the overflow issue. But I couldn't find the general form where we could use a $d$ dimensional $x$. – PickleRick Mar 14 '17 at 09:49
  • @Xi'an if it was only one Gaussian, then the log-likelihood would have been easily computed as [here](https://www.statlect.com/images/normal-distribution-maximum-likelihood__10.png) – PickleRick Mar 14 '17 at 10:03
  • 1
    @Xi'an thank you for your time, and your suggestion! your answer seems logical. I am going to put it to work and see how it performs... – PickleRick Mar 14 '17 at 10:07
  • 1
    @Anoosh: can you tell which part in the density is responsible for the underflow: is it the determinant $|\Sigma_c|$ or the quadratic form $(x_t-\mu_c)'\Sigma^{-1}_c(x-\mu_c)? – Xi'an Mar 14 '17 at 10:43
  • @Xi'an actually it's both. But the inverse covariance in the quadratic form $(x_t-\mu_c)'\Sigma^{-1}_c(x-\mu_c)$ is most of the time the problem. Inverse covariance gets huge when the covariance is small, then exponential of it is even bigger. If we only limit one (quadratic form or $|\Sigma_c|$), then when they multiply to eachother, again the overflow might happen. – PickleRick Mar 14 '17 at 10:46

1 Answers1

3

Switching logs and sums is not a possibility. When considering$$\sum_{t=1}^t \log\left[ \sum_{c=1}^C w_c p(x_t|\mu_c,\Sigma_c) \right]$$from an overflow-underflow perspective, you could

  1. ascertain whether or not exploring such extreme values for your parameters makes sense for the problem at hand. If not, treat extreme values as producing zero likelihoods;
  2. check if any of the determinants $|\Sigma_c|$ creates an indeterminacy issue and if so treat the determinant aside from the rest of the density by computing directly the log;
  3. determine which term $w_c p(x_t|\mu_c,\Sigma_c)$ is largest among all pairs $(t,c)$. For this you can look directly at the logarithms since$$\arg\max_{t,c} w_c p(x_t|\mu_c,\Sigma_c) =\arg\max_{t,c} \log\{w_c p(x_t|\mu_c,\Sigma_c)\}$$If this maximum creates an indeterminacy, then treat the whole likelihood as zero;
  4. remove this maximal value from all terms and treat underflow terms as zeros. If a given observation $x_t$ is such that all (log-)terms $\log\{w_c p(x_t|\mu_c,\Sigma_c)\}$ are indeterminate, then treat the whole likelihood as zero;
  5. check this impressive answer on Cross validated about the approximation of a Gaussian tail density by W. Huber if the problem persists.
Community
  • 1
Xi'an
  • 90,397
  • 9
  • 157
  • 575