5

When reading the topic modeling tutorial written by Blei, KDD 2011 tutorial I was confused about a set of diagrams which aim to show the effect of $\alpha$ in Dirichlet distribution.

For example, for the plot with $\alpha=1$, what am I suppose to discover? What does item mean here? Do those 15 items mean a 15-dimensional probability vector? The $\alpha=1$ is assumed to lead to a uniform distribution. But I feel confused about how to connect this plot with a uniform distribution. I also list other plots with $\alpha=100$ and $\alpha=0.01$.

enter image description here

enter image description here

enter image description here

Alexis
  • 26,219
  • 5
  • 78
  • 131
user3269
  • 4,622
  • 8
  • 43
  • 53

1 Answers1

3

The x-axis are group assignments and the y-axis is the corresponding probability.

$\alpha$ is the prior controlling how much you weigh previously selected groups when selecting a new group assignment.

As $\alpha$ gets smaller you weigh previously selected groups more heavily, hence for $\alpha=0.1$ only a few groups are selected. As $\alpha$ gets larger you weigh the previously selected groups less and less, hence the uniform distribution of groups for $\alpha=100$.

Note $\alpha=1$ corresponds to a uniform prior for the number of groups, but the resulting distribution will not be uniform. In general, larger $\alpha$ equals more groups, smaller $\alpha$ equals less groups.

Glen
  • 6,320
  • 4
  • 37
  • 59
  • Thanks! Could I understand the graph this way. These plots related to a scenario with 15 items and 10 groups. Using $\alpha=0.1$ as an example, here item 1 is more likely to be assigned to group 7 with pmf almost equal to 1; while item 2 more likely to be assigned to group 9 and group 6, where group 9 with pmf almost equal to 0.8 and group 6 with pmf almost equal to 0.2. – user3269 Apr 30 '14 at 03:26
  • No I don't think this is correct. Look at it as 15 simulations of 10 possible groups (number of items is unknown). In the first simulation, almost all items were assigned to 7. In the second simulation, almost all items were assigned to 6 and 9. And so on... – Glen Apr 30 '14 at 16:25