6

After knowing how LSA works, I went on continue reading on pLSA but couldn't really make sense of the mathematical formula. This is what I get from wikipedia (other academic papers/tutorial show similar form)

\begin{align} P(w,d) & = \sum_{c} P(c) P(d|c) P(w|c)\\ & = P(d) \sum_{c} P(c|d) P(w|c)\\ \end{align}

I gave up trying to derive it, and found this instead

\begin{align} P(c|d) & = \frac{P(d|c)P(c)}{P(d)}\\ P(c|d)P(d) & = P(d|c)P(c)\\ P(w|c)P(c|d)P(d) & = P(w|c)P(d|c)P(c)\\ P(d) \sum_{c} P(w|c)P(c|d) & = \sum_{c} P(w|c)P(d|c)P(c) \end{align}

How does the summation appear at the last line? I am currently reading through some tutorial on Bayesian Inferencing (learnt basic probability rules and Bayesian theorem before but can't really see them being useful enough here).

Jeffrey04
  • 187
  • 9

2 Answers2

6

I am assuming you want to derive:

\begin{align*} P(w,d) = \sum_{c} P(c) P(d|c) P(w|c) &= P(d) \sum_{c} P(c|d) P(w|c) \end{align*}

Further, this is similar to Probabilistic latent semantic indexing (cf. Blei, Jordan, and Ng (2003) Latent Dirichlet Allocation. JMLR section 4.3). PLSI posits that a document label $d$ and a word $w$ are conditionally independent given an unobserved topic $z$.

If this is true, your formula is a simple consequence of Bayes theorem. Here are the steps:

\begin{align*} P(w, d) &= \displaystyle \sum_z P(w, z, d)\\ & = \displaystyle \sum_z P(w, d | z) p(z)\\ &= \displaystyle \sum_z P(w | z) p(d|z) p(z), \end{align*} where factorization into products is because of conditional independence.

Now use Bayes theorem again to get \begin{align} \displaystyle \sum_z P(w | z) p(d|z) p(z) &= \displaystyle \sum_z P(w | z) p(z,d)\\ &= \displaystyle \sum_z P(w | z) p(z|d)p(d)\\ &= p(d)\displaystyle \sum_z P(w | z) p(z|d) \end{align}

mpiktas
  • 33,140
  • 5
  • 82
  • 138
suncoolsu
  • 6,202
  • 30
  • 46
  • @Jeffrey04 pls let me know if pLSA and pLSI are different. – suncoolsu Mar 28 '11 at 11:25
  • 1
    @suncoolsu, I edited the latex code for better readability. If it is not ok, please feel free to rollback. – mpiktas Mar 28 '11 at 11:53
  • @mpiktas. Thank you; I did not know how to format LaTeX code here. You are a great moderator! – suncoolsu Mar 28 '11 at 12:11
  • @suncoolsu, it is actually not different from the usual latex code. The only difference that slashes need to be doubled sometimes. You can view the source of any latex formula by right clicking on it by the way. – mpiktas Mar 28 '11 at 12:14
  • @suncoolsu pLSA/pLSI should be similar, it is just that IR community prefers the term pLSI iirc – Jeffrey04 Mar 29 '11 at 01:40
  • erm... why is there a summation sign when you derive from P(w,d) to P(w,z,d)? – Jeffrey04 Mar 29 '11 at 03:18
  • @jeffrey. think about it as integrating it out. as $z$ is discrete so you want to sum it out. – suncoolsu Mar 29 '11 at 09:24
3

The line $P(c|d)P(c) = P(d|c)P(c)$ (your eq 2) should be $P(c|d)P(d) = P(d|c)P(c)$.

I'm not sure why you don't think Bayes theorem and basic probability rules are useful:

Eq 1 is Bayes theorem (ie recognizing that $P(d|c)P(c) = P(c,d)$ and plugging in to the definition of conditional probability)

Eq 2 follows immediately from eq 1

Eq 3 is just eq 2 multiplied through by $P(w|c)$.

Since eq 3 holds for all $c$ the sums are equal. Then since $w$ is independent of $d$ given $c$ (an assumption from the model), $P(w|c)P(c|d) = P(w|c, d)P(c|d) = P(w,c|d)$ and so $\sum_c\ P(w,c|d) = P(w|d)$ by the law of total probability, giving you $P(w|d)P(d)$.

Finally, $P(w|d)P(d)=P(w,d)$ from the definition of conditional probability.

So basic probability is in fact both necessary and sufficient for the derivation!

JMS
  • 4,660
  • 1
  • 22
  • 32
  • erm... i meant knowing bayes theorem and basic probability rules do not seem enough to be able to derive the formula... sorry for the confusion – Jeffrey04 Mar 29 '11 at 01:39
  • @Jeffrey04 No worries; hopefully between @suncoolsu's answer and mine it's clearer. – JMS Mar 29 '11 at 02:07