5

I've been watching a talk (section between 07:20-08:00) given by Michael Jordan and I'm getting confused between independence and exchangeability.

He says that

"If we have a document and you believe that it's exchangeable, i.e. you can move the words around and it doesn't matter, you don't mean iid. In particular, if you see the first word and it's an Italian word, I think it's likely that the second word will also be an Italian word too, and under iid that wouldn't be the case."

I understand that bag-of-words model assumes exchangeability, but why are words not independent? Doesn't observing one word not affect the probability distribution of the second word?

user48935
  • 111
  • 3
  • This may help: http://stats.stackexchange.com/questions/34465/what-is-so-cool-about-de-finettis-representation-theorem – Zen Sep 04 '16 at 04:38
  • 1
    If I observe the word "the", are you saying that this ought to have no effect on the probability distribution of the next word to appear? – shadowtalker Sep 04 '16 at 11:22
  • @ssdecontrol, No. I think I got confused about the context in which Michael Jordan was speaking. I was thinking of the case where the entire collection of documents and their words are fully observed, in which case, picking a particular word at a time step does not change the distribution for the next word in the next time step. Thanks for your input though. – user48935 Sep 20 '16 at 19:09

1 Answers1

2

As I understand it, what he's saying is this:

Suppose we have a bag of words and we know only that it is exchangeable. What he's saying is that just because we know it's exchangeable, doesn't imply that it's also independent and identically distributed (iid). To show this he gives you a simple counter example:

We have a bag of words and we know it's exchangeable. But we then observe the first word and see it's Italian. Now, given that we know a word in this document is Italian, and that documents are generally written in only one language, we would expect the probability that the remaining words are also Italian is higher than other languages. So loosely speaking the words are not independent since observing one word influences the probability distributions of the remaining words.

Before any word is observed, all languages are possible, and so any word is possible. However, once we observe one word, then (generally speaking ofcourse) that document should only be written using a language that word belongs to, and so words which do not belong do that language are no longer possible, meaning observing one word has changed the probability distribution of the remaining words, and hence they aren't independent.

Exchangeable requires words to be identically distributed, but it does not require independence. Hence, if we know the document is exchangeable that doesn't mean we can assume it's independent.

Patty
  • 1,599
  • 1
  • 8
  • 21