How do I calculate a Maximum Likelihood Parameter Estimate for this binary data - Naive Bayes

Question

I find I best learn by example but I can't seem to find any that match with this, or at most, people appear bizarrely unwilling to show where in these abstract equations you actually insert which numbers to do an actual calculation. Or make some assumptions like the domain being Gaussian that doesn't seem to apply to what I'm doing.

So I have a table of binary training data, that is, 3 input columns which I'll label x1, x2, and x3, and an output column which I'll call C. All of these simply contain yes/no (or 1/0 if you prefer) answers. Feel free to redefine these names if they collide with your preferred notation.

Here is an example table of data, I suspect this will not format nicely on mobile devices.

+-----+-----+-----+-----+
| x1  | x2  | x3  |  C  |
+-----+-----+-----+-----+
| yes | yes | no  | no  |
| no  | yes | no  | yes |
| yes | yes | no  | no  |
| no  | yes | no  | no  |
| yes | no  | yes | yes |
| no  | yes | no  | yes |
| no  | yes | no  | no  |
| yes | no  | yes | yes |
| yes | yes | no  | yes |
| no  | yes | no  | no  |
| yes | no  | no  | yes |
+-----+-----+-----+-----+

$P(C=y) = 6/11$, $P(C=n) = 5/11$

What should I do with this to produce some sort of MLE? What does that tell me about my data? Am I missing the point?

I have seen this CV.SE question, and I feel like it's not a big step to apply its accepted answer to my data but I just can't connect it with this data set, I need to see an example of it being used.

Thank you very much. :) If there's any further info you need do comment and ask, I'm new to both CV and fairly new to this sort of thing in general

Answer 1

The goal is to obtain the conditional distribution $P(C|x_1, x_2, x_3)$, which we model as \begin{align*} P(C|x_1, x_2, x_3) &= \frac{P(x_1,x_2,x_3|C)P(C)}{P(x_1,x_2,x_3)} && \text{(Bayes rule)} \\[1.2ex] &= \frac{P(x_1|C)P(x_2|C)P(x_3|C)P(C)}{P(x_1,x_2,x_3)} && \text{(the Naive Bayes assumption)} \\[1.2ex] &= k P(x_1|C)P(x_2|C)P(x_3|C)P(C) \end{align*} The constant $k$ can be ignored, and computed later, by noting that $$P(C=\text{yes}|x_1,x_2,x_3) + P(C=\text{no}|x_1,x_2,x_3) = 1.$$

---

Application to toy data

First we have $$P(C=\text{yes}) = \frac{6}{11} \quad\quad P(C=\text{no}) = \frac{5}{11}.$$

The distribution of each $x_i$, conditional on $C$ can also be obtained from the data.

$$P(x_1=\text{yes}|C=c) = \begin{cases} \frac{4}{6}, & c=\text{yes} \\[1.1ex] \frac{2}{5}, & c=\text{no} \end{cases}$$

$$P(x_2=\text{yes}|C=c) = \begin{cases} \frac{3}{6}, & c=\text{yes} \\[1.1ex] \frac{5}{5}, & c=\text{no} \end{cases}$$

$$P(x_3=\text{yes}|C=c) = \begin{cases} \frac{2}{6}, & c=\text{yes} \\[1.1ex] \frac{0}{5}, & c=\text{no} \end{cases}$$

Now you would like to predict the class of a new instance with the data ($x_1=\text{yes}, x_2=\text{yes}, x_3=\text{yes}$). To do this, compute

$$P(C=\text{yes}|x_1=\text{yes}, x_2=\text{yes}, x_3=\text{yes}) = k\frac{4}{6}\frac{3}{6}\frac{2}{6}\frac{6}{11} = k(0.0606)$$ $$P(C=\text{no}|x_1=\text{yes}, x_2=\text{yes}, x_3=\text{yes}) = k\frac{2}{5}\frac{5}{5}\frac{0}{5}\frac{5}{11} = 0$$

Indicating that the the class is yes with probability one. The problem is, that with such a small data set, you will see some $0$s and $1$s in the conditional distributions $P(x_i|C)$. To account for this, we can use Laplace smoothing which involves adding $\epsilon$ to the numerator and $d \epsilon$ to the denominator (where $d=2$ is the number of classes) of the conditional distributions. Setting $\epsilon = 1$ we would obtain

$$P(C=\text{yes}|x_1=\text{yes}, x_2=\text{yes}, x_3=\text{yes}) = k\frac{5}{8}\frac{4}{8}\frac{3}{8}\frac{6}{11} = k(0.0639)$$ $$P(C=\text{no}|x_1=\text{yes}, x_2=\text{yes}, x_3=\text{yes}) = k\frac{3}{7}\frac{6}{7}\frac{1}{7}\frac{5}{11} = k(0.0239)$$

Choosing $k$ so that these sum to $1$, we get that this instance is class "yes" with probability $0.73$.

Answer 2

This answer assumes you only want to estimate the parameters of a model that predicts $C$ from $x_i$.

We are modeling a conditional distribution $P(C|x_1,x_2,x_3)$.

The Naïve Bayes (NB) assumption is a conditional independence of predictors given the result: $P(x_1,x_2,x_3|C) = \prod_i{P(x_i|C)}$.

This can be used to calculate the goal distribution using Bayes theorem: $P(C|x_1,x_2,x_3) = P(x_1,x_2,x_3|C)P(C)/P(x_1,x_2,x_3)$.

Combining the two equations, we get: $P(C|x_1,x_2,x_3) = P(C)/P(x_1,x_2,x_3)\prod_i{P(x_i|C)}$.

This model has these parameters to be estimated: $P(C)$ and $P(x_i|C)$.

Technically, there is also $P(x_1,x_2,x_3)$, which you need in order to normalize and get actual conditional probability. If you are only interested in prediction, you can get away with comparing non-normalized likelihoods, then you do not need this parameter. Another way to see this is modeling a joint distribution instead: $P(C,x_1,x_2,x_3) = P(C)\prod_i{P(x_i|C)}$.

The MLE for these parameters is obtained simply by taking the ratio of counts (as you did, at least for some).

TLDR: At the moment you divided counts, you performed MLE, and if that was all required, you are done.

If, however, you want to perform predictions, you can calculate either conditional probabilities (more computation) or likelihoods (and compare them or take their ratio).