0

In Goodfellow's Deep learning text, it is written

enter image description here

Is this way of defining a probability $p(y=1| x;\theta)$ even legal?

Recall the definition of a probability given a random variable

enter image description here

where $p_X$ is the probability mass function.

Is the logistic function $\sigma$ considered to be a probability mass function?

If so, what plausible process is responsible for generating this probability mass function? (i.e., a process similar to how geometric or Bernoulli pmf are defined)

The man of your dream
  • 171
  • 8

1 Answers1

1

Imagine your data is divided in two classes $C_{1}$ and $C_{2}$. Regardless of the probability distribution you may assume (I get to that later), using Bayes theorem, you have, $$ P(C_{1}|x) = \frac{P(x|C_{1})}{P(x|C_{1})P(C_{1}) + P(x|C_{2})P(C_{2})} = \frac{1}{1+exp(-a)} $$ where, $$ a = ln\frac{P(x|C_{1})P(C_{1})}{P(x|C_{2})P(C_{2})} $$ which is the sigmoid. The interesting this now is that, it reduces to linear models when working with generalized linear models (assuming you model all conditional distributions with the same family -Gaussian, binomial, Poisson,...).

jpmuc
  • 12,986
  • 1
  • 34
  • 64
  • your first equation is incorrect and $a$ is not the sigmoid function. Also, I'm not sure what mentioning Bayes' theorem adds to this. After all, logistic regression does not use Bayes theorem, whereas some of its competitors do, e.g. Naive Bayes, LDA, QDA, etc. – Taylor Jan 20 '19 at 23:06