0

In a machine learning algorithm that I'm using, I need to get the exponential values of something in one of the steps.

This is the step that I'm dealing with right now: enter image description here

I've already got all the 1+g_j(X_i) etc etc calculated, there's no problem in that. Let's call it calculated_value. That's fine.

The problem is, I am getting infinities and overflows when implementing this function in code.

What's the solution for this? What would the logarithmic version of the same function, that I could use in place of the function above?

In case it matters, β is going to be used as a coefficient to some variables, including partially to calculate the weights for a weighted regression solution.

user961627
  • 223
  • 1
  • 2
  • 10
  • A very good approximation for $\ln(1+\exp(x))$ for $x>7$ or so is $x+\exp(-x)$. This means you can readily take logs of numerator and denominator and work on log-scales when dealing with large arguments. – Glen_b May 08 '14 at 03:46

1 Answers1

1

One way to handle this would be to rescale the terms in the numerator and denominator by a suitably large constant $C$, which is equivalent to subtracting $log(C)$ from the numerator of each exponential terms as follows. I'll reduce the notation of the problem for simplicity.

$\beta_j(X_i) = \frac{e^A}{1 + e^A + e^B}= \frac{ C^{-1}e^A}{C^{-1} + C^{-1}e^A + C^{-1}e^B} = \frac{ e^{A-log(C)}}{C^{-1} + e^{A-log(C)} + e^{B-log(C)}}$

One possible choice would be to let $log(C) = max(A,B)$. Then at least you are guaranteed to not have overflow!

jsk
  • 2,810
  • 1
  • 12
  • 25
  • Is the formula that you're proposing a kind of "normalization"? (I'm still learning here..) – user961627 May 08 '14 at 10:51
  • In fact, even the function I asked about above, that too is a kind of normalization, right? – user961627 May 08 '14 at 11:30
  • 1
    Sure, you could call it that. The function you asked about is itself a normalization, but the normalization fails if any one of the exp(x) is inf! – jsk May 08 '14 at 15:42