What's the difference between estimating on a dataset $P(X|Y)$ and $P(Y)$ vs $P(Y|X)$?

Question

In chapter 3 of his excellent book ("Generative and discriminative classifiers: Naive Bayes and logistic regression") , Tom Mitchell says that, when learning classifiers based on Bayes rule, one can estimate the parameters of $P(X|Y)$ and $P(Y)$ by maximum likelihood. Example: for discrete $Y$ and $X$, estimator for $P(X=x_i|Y=y_j)$ would be the proportion in the dataset of $X$ having value $x_i$ when $Y$ has value $y_j$. Estimator for $P(Y=y_j)$ would be the proportion of $Y$ having value $y_j$.

Thing that I don't understand is "what is the difference between 1) estimating $P(X|Y)$ and $P(Y)$ parameters as stated above, make the product and divide by $P(X)$ and 2) estimating directly the parameters of $P(Y|X)$? For example, estimator for $P(Y=y_j|X=x_i)$ would be the proportion of $Y$ having value $y_j$ when $X$ has value $x_i$. The result is the same, isn't it?

Am I missing something?

Answer 1

As other people have pointed out, in your particular example, whether you calculate $P(Y | X)$ directly from your data or you use Bayes Theorem it makes no difference. Now you posed the following question

I still don't understand why I would estimate the parameters using the "long way" (see my comment in answer to @dnqxt). Suppose I have a relatively small sample of patients for which I have the information about having cancer or not and having tested positive or not on the test. Why estimate $P(C|+)$ by calculating $P(+|C)P(C)P(+|C)P(C)+P(+|\bar{C})P(\bar{C})$ when I can estimate it directly by calculating P(C|+) on my sample?

The answer is, you wouldn't use Bayes Theorem for this problem because there is no need. The canonical problem where Bayes Theorem shows up is when you aren't given the actual data, but you are given estimates of the probabilities. Usually it's stated like "Suppose the prevalence of this disease in the population is $p_1$, and the test has a sensitivity of $p_2$ and a specificity of $p_3$ etc. You test positive. What's the probability you actually have the disease. Without any other information, the only way to proceed is with Bayes Theorem

The other place that Bayes theorem shows up is hinted at in the title of that book chapter, namely in Naive Bayes, where you have some categorical outcome $Y$ and you want to correctly classify which group it belongs to based on some arbitrary feature vector $X = (X_1, \dots, X_n)$. If the features are all binary or categorical, then with enough data you again don't need to use Bayes theorem because you can estimate $P(Y|X)$ directly (at least in theory). But when you start throwing in continuous features things get more difficult, and one way to proceed is by using Bayes theorem to write it as

$P(Y | X_1, \dots, X_n) \propto P(X_1, \dots, X_n | Y)P(Y)$,

where $\propto$ denotes proportionality. You then make some assumptions about $P(X_1, \dots, X_n | Y)$ and $P(Y)$ to make the calculation simpler (the "naive" part of naive Bayes) and voila!, you have a naive bayes classifier. See https://en.wikipedia.org/wiki/Naive_Bayes_classifier for more info

Answer 2

You are right, no matter if you estimate $P(Y|X) = \frac{P(X|Y)\,P(Y)}{P(X)}$ using Bayes theorem, or estimate $P(Y|X)$ directly from the data, the result would be the same. If it is not the same, then something is wrong (you made mistake, it is a problem of computer numerical precision etc.). We use Bayes theorem in cases when we can't calculate the probability directly, or for other reasons, doing this indirectly has advantages. For example, you can check this thread, where it is described in detail how Bayes theorem is applied to medical data. In such case we can conduct a case-control study and learn how likely medical test is to discover the condition $P(+|C)$, we also may know what is the prevalence of the condition $P(C)$, but if the condition is rare, it would be insufficient to sample huge group of test takers, just to see how many of them has the condition $P(C|+)$.

For example, say that prevalence of the disease is $P(C)=0.0005$. Let's assume that probability of taking the medical test for the disease is independent of having this disease, so we can expect same proportion of infected people among those who took the test as in the general population. This means that among test-takers, you would need to sample $2000$ people to expect just a single person affected by the disease. What follows, overall size of sample needed to study $P(C|+)$ would need to be huge. On another hand, you could simply take a group of patients who have this condition (e.g. at the hospital) and controls who don't, and test them using your test. This would enable you to calculate $P(+|C)$, and then use Bayes theorem to get the "reverse" conditional probability. (Notice that this is a simplified example, I do not claim that this is a realistic example of how medical trials are conducted.)

Answer 3

Consider the following simple data set consisting of a list of pairs representing (X,Y) tuples:

$(0,0),(0,0), (0,1), (1,0), (1,1).$

Then the estimate of $P(X=0|Y=1) = 1/2$, which is different from the estimate of $P(Y=1|X=0) = 1/3$