8

There are a lot of places where you'll see the proof that Naive Bayes classifiers are linear, like this and this. But they always assume a special case of the family of Naive Bayes classifiers which more often than not happens to be Multinomial Naive Bayes.

$$ P(X_1 = x_1, X_2 = x_2, \cdots , X_k = x_k) = \frac{n!}{x_1!x_2!\cdots x_k!} p_1^{x_1} p_2^{x_2} \cdots p_k^{x_k} $$

Now, Multinomial Naive Bayes can be proved to be a linear classifier as follows:

Considering a binary classification example with $y \in \{0, 1\} $, the decision boundary would be:

$$ \begin{equation} log(\frac{P(Y = 1|X = x)}{P(Y = 0|X = x)}) = 0 \end{equation} $$

where $X = (X_1, X_2, \cdots ,X_k)$ and $x = (x_1, x_2, \cdots ,x_k)$

Using Bayes' rule and multinomial distribution, the decision boundary could be written as:

$$ \sum_{i=1}^{k} x_i log(\frac{p_i}{p_i^{'}}) + log(\frac{P(Y=1)}{P(Y=0)}) = 0 $$

where $p_i$ and $p_i^{'}$ are probabilities of the occurrence of ith feature in class $y = 1$ and $y = 0$ respectively.

So, we've got a linear form ($w^Tx + b = 0$) for our decision boundary.

But is this true for the entire family of Naive Bayes classifiers?

I suspect it would depend on the the nature of the probability distributions you choose for the input features.

Bharat Khatri
  • 295
  • 2
  • 10
  • 1
    I was thinking about this recently. I wrote a related answer earlier (on naive bayes and discriminant analysis methods) -- https://stats.stackexchange.com/a/297877/1704 -- but I didn't look into whether all the "Discriminant Analysis" methods were linear. I know that QDA is *not* linear. And, as you mentioned, the standard discrete/multinomial NB is linear). Need to think about the others. A quick experiement had GNB (gaussian NB) non-linear, but I'm not convinced I didn't make a mistake. – MrDrFenner Mar 27 '18 at 12:28

0 Answers0