Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [underflow]

14 questions
16
votes
3 answers

Example of how the log-sum-exp trick works in Naive Bayes

I have read about the log-sum-exp trick in many places (e.g. here, and here) but have never seen an example of how it is applied specifically to the Naive Bayes classifier (e.g. with discrete features and two classes) How exactly would one avoid the…
Josh
  • 3,408
  • 4
  • 22
  • 46
9
votes
1 answer

Kernel density estimation on bounded support?

I was looking for some way to deal with boundary bias of kde in case of a unit interval. One example is an usage of Chen estimators (or Beta estimators; an example might be seen here: http://stats-www.open.ac.uk/TechnicalReports/mcjdah.pdf -p.4) -…
Tzab
  • 91
  • 3
6
votes
2 answers

Subtracting very small probabilities - How to compute?

This question is an extension of a related question about adding small probabilities. Suppose you have log-probabilities $\ell_1 \geqslant \ell_2$, where the corresponding probabilities $\exp(\ell_1)$ and $\exp(\ell_2)$ are too small to be…
Ben
  • 91,027
  • 3
  • 150
  • 376
3
votes
1 answer

Adding very small probabilities - How to compute?

In some problems, probabilities are so small that they are best represented in computational facilities as log-probabilities. Computational problems can arise when you try to add these small probabilities together, since some computational…
Ben
  • 91,027
  • 3
  • 150
  • 376
2
votes
1 answer

Computation within log space

What is the conversion of the following equation into log space? $bf2 = 1 + (p * (bf1 - 1))$ Given log.bf1 (log Bayes factor), how do I get to log.bf2 without having to compute bf1, but instead using log.bf1? Context: Numerical issues are arising…
user3302113
  • 23
  • 3
2
votes
1 answer

Vectorised computation of logsumexp

In this related post there is an explanation of how you can add together two very small probabilities using the logsumexp function, and how this can be programmed into base R. How can this method be extended to get a general vectorised function…
Ben
  • 91,027
  • 3
  • 150
  • 376
2
votes
1 answer

How to randomly sample values given the extremly small and large log-probabilities?

Assume a long list of log-values. The list consists of very small negative numbers, very large negative numbers, as well as very large positive numbers. To avoid numerical overflow/underflow, I need to stay in log space. Now the question is, how can…
user3639557
  • 1,042
  • 10
  • 21
1
vote
0 answers

Avoiding underflow with arbitrarily large number of data points in a joint likelihood calculation?

I am trying to compute a Bayesian posterior distribution, given a large number of data points. I’ve found that as the number of data points increases, the joint likelihood of the data underflows to zero. The common solution to this problem is to…
Ravi P.
  • 11
  • 2
1
vote
0 answers

Insufficient floating point precision for the correct computation of a density

I'm using the Metorpolis-Hastings algorithm in a setting where the acceptance function is essentially of the form $$\alpha(x,y)=1\wedge\frac{u(x,y)}{v(x,y)},$$ where $$u(x,y)=p+(1-p)\prod_{i=1}^mu_i(x,y)$$ and…
0xbadf00d
  • 539
  • 3
  • 8
1
vote
0 answers

Underflow when estimating marginal likelihood via bridge sampling

I try to use an iterative procedure to estimate the marginal likelihood in a Bayesian setting for model selection. In case you are interested in the specifics of bridge sampling in my application, see this or this paper for instance. I'm fighting…
yrx1702
  • 642
  • 4
  • 16
1
vote
0 answers

Sum of very low probability

I have a score of some feature, $F_1$ and $F_2$ where this score is the logarithm of probability. This score is very low and very sparse, for example i have: $F_1$ with score $-800$ $F_2$ with score $-3000$ I need to get the probability with the…
Neptune
  • 579
  • 3
  • 15
0
votes
0 answers

Multivariate Normal Underflow

Good day everyone At the moment I am attempting to write code in R to calculate the following. $$ \tau_{k j}^{(m)}=\frac{\pi_{k}^{(m)} f_{k}\left(x_{j} ; \theta_{k}^{(m)}\right)}{f\left(x_{j} ; \Theta^{(m)}\right)}=\frac{\pi_{k}^{(m)}…
Susan-l3p
  • 83
  • 5
0
votes
0 answers

Equivalent of log sum exp trick for subtraction

I have two small positive real numbers $u, w$ such that $u > w$. Given $\log(u), \log(w)$ I'd like to find a numerically stable way to calculate $\log(u - w)$. One possible way of transforming the preceding is : $\log (u - w) = \log(u) + \log(1 -…
marcusy
  • 33
  • 3
-1
votes
1 answer

Avoiding Matlab's underflow prevention in backprop, due to performance cost

In Matlab, I understand that if a number gets closer to zero than realmin, then Matlab converts the double to a denorm . I am noticing this causes significant performance cost. In particular I am using a gradient descent algorithm that when near…
rnoodle
  • 203
  • 2
  • 13