1

From what I understand, MLE for a model helps us find out what parameters in the model will suit the data most.

Thus, in linear regression, we try to find $L(\theta)$ as $\prod_{i=1}^{m}p(y|x;\theta)$ i.e., we try to find what $\theta$ will suit the data points most and we already have $x$ given to us. This is similar to logistic regression, where also we maximize $\prod_{i=1}^{m}p(y|x;\theta)$. However, for GDA and Naive Bayes, we define $L(\theta)$ as $\prod_{i=1}^{m}p(y,x;\theta)$ because these are generative models and we find both $x$ and $y$ ourselves.

Is my understanding correct? If not, then why is MLE calculated differently for GDA and linear regression?

skyquake
  • 95
  • 1
  • 1
  • 5

1 Answers1

2

Notice that for logisitc regression $p(y|x;\theta)$ is a Bernoulli distribution parametrized by mean $\theta$ that depends on $x$, while in naive Bayes algorithm $p(y, x)$ is the joint empirical distribution of $x$ and $y$ that is used to obtain the probabilities $\theta$. Naive Bayes algorithm estimates $\theta$ directly from the data, using empirical distributions (or their approximations, like in the case of Gaussian naive Bayes algorithm).

Recall that $p(y|x) = p(y, x) \;/\; p(x)$, by the properties of conditional probability, where $p(x)$ is just another empirical probability distribution over $x$. In naive Bayes algorithm, we obtain $\theta = p(y, x) = p(x | y) \,p(y) $. Calculating $p(x)$ to get $ p(y | x) = p(y, x) \;/\; p(x) $ does not change anything about the ordering of probabilities, since if you had $\Pr(Y=0 , X = x) \;/ \; \Pr(X = x)$ and $\Pr(Y=1 , X = x) \;/ \; \Pr(X = x)$ would be divided by the same constant $\Pr(X = x)$, so we can abandon the useless computation. By doing so we are calculating the same thing, but more efficiently. You can calculate it and obtain the conditional probability $p(y|x; \theta)$, but it does not change anything about the problem.

Tim
  • 108,699
  • 20
  • 212
  • 390