4

I understand that the normal distribution is undefined if the standard deviation is zero, but I need to handle the case where all values are equal in a computer algorithm. The following method must return a valid value, even if the standard deviation is zero. How can I fix this method so it does not divide by zero?

public static double NormalDist(double x, double mean, double standard_dev)
{
   double fact = standard_dev * Math.Sqrt(2.0 * Math.PI);
   double expo = (x - mean) * (x - mean) / (2.0 * standard_dev * standard_dev);
   return Math.Exp(-expo) / fact;
}

My idea was to insert this at the beginning of the method:

        if (standard_dev == 0.0)
        {
            return x == mean ? 1.0 : 0.0;
        }

Would this be correct?

user130307
  • 43
  • 1
  • 1
  • 3
  • 2
    You didn't "fix" it.The PDF of normal distribution is undefined for $\sigma=0$, it's not 1 like in your "fix" – Aksakal Sep 07 '16 at 16:54
  • 1
    @Aksakal points to an important issue: exactly how do you intend to use `NormalDist` in your algorithm? – whuber Sep 07 '16 at 17:05
  • This is being used in a naïve bayes classifer. It's attempting to select a class based on data in a training set. To me, if seems that if all values in a set of parameters are the same, they should have no influence on the outcome. So, for example, if we are looking at how shoe size falls within a distribution to determine if a person (the class) is male or female, but if it just so happens that all persons in the distribution have exactly the same shoe size, then shoe size cannot be used to differentiate and should be removed from consideration. I am looking to handle this case. – user130307 Sep 07 '16 at 18:32
  • 1
    Maybe the answer is not in changing the output of this method, but rather to just throw out that value altogether. – user130307 Sep 07 '16 at 18:36

2 Answers2

4

When standard deviation is zero, your Gaussian (normal) PDF turns into Dirac delta function. You can't simply plug zero standard deviation into the conventional expression. For instance, if the PDF is plugged into some kind of numerical integration, this won't work. You have to modify the integrals. In the example below we calculate the mean value of function $g(x)$ using the Gaussian density $f(x|\mu,\sigma^2)$:

$$\int g(x)f(x|\mu,\sigma^2)dx$$

when you plug zero variance, this becomes delta-functional: $$\int g(x)f(x|\mu,0)dx=\int g(x)\delta(x-\mu)dx=g(\mu)$$

Your code has to be able to recognize this, otherwise it'll fail.

One way to fix this is surprisingly simple: plug a very small value of $\sigma$ into Gaussian instead of zero. You'll have to pick the right $\sigma$ for your situation. If it's too small then it'll blow up your exponent, and the integrals will not work or the precision will be low. This goes to a known Gaussian approximation of delta function: $$\delta(x)=\lim_{\sigma\to 0}\mathcal N(0,\sigma)$$

Aksakal
  • 55,939
  • 5
  • 90
  • 176
  • 1
    Whether it "fails" has to depend on its functional specification. If the intention is merely to return a continuous version of the probability density, then the function that always returns zero is correct, because the Dirac delta has a density defined everywhere but at the mean. If, on the other hand, this function is being called as part of a likelihood calculation for computing an ML estimate, then there is no implementation whatsoever that will cause it to succeed: all it can do is raise an error condition. – whuber Sep 07 '16 at 18:25
  • @whuber, floating [point format](https://en.wikipedia.org/wiki/IEEE_754-1985#Positive_and_negative_infinity) supports positive infinity. So, he could set the PDF to infinity. The problem's that even then his likelihood optimization routine will not work, because of the discontinuity. Hence, I'm saying that he has to modify the code that uses PDF. He can't get away with just messing with PDF alone – Aksakal Sep 07 '16 at 18:42
  • I posted an additional comment above with more info. I'm thinking Aksakal is correct. I have to modify the code that uses the method. – user130307 Sep 07 '16 at 19:07
1

This is a question in the Statistics textbook by Hogg and Craig! The authors give a hint: Look at the moment generating function of the normal and plug in sigma = 0.

So before going to the answer, let's remember why this works - moment generating functions are unique.

The moment generating function of the normal, N(a,b^2), M(t|a,b^2) =

exp(at + t^2 * b^2 /2)

Setting b=0, we have

M(t|a,B^2) = exp (at)

This is the moment generating function of

f(x) = a. (I wouldn't call this a Dirac Delta function, I would call this a constant. Note the Dirac Delta function is not technically a PDF.)

This result shouldn't be a surprise. As the variance decreases, the probability gets closer to the mean, and so the limiting distribution is the function equals the mean.

Of course this can be proven directly; we can look at a sequence of functions fn(x) = N(a,b^2/n) and we see the sequence has the variance approach 0 as n approaches infinity. Seeing it converges in probability to the constant is pretty easy; you can show it converges almost surely but this will take a little more work.

But that isn't the exact question - which can be answered as Hogg and Craig suggested!

Breunor
  • 11
  • 1
  • Please reconsider the distinction between $\delta$ and a constant: it's *huge.* At https://stats.stackexchange.com/a/73626/919 I explain some of the differences. – whuber Feb 03 '22 at 18:33