4

I have a question regarding Latent Semantic Analysis - after performing SVD decomposition of term-document matrix and choosing some number of dimensions, I get the set of new document vectors.

Now, how can I calculate similarity between two documents? New document vectors contain negative values, and results produced by cosine similarity make no sense.

user1315305
  • 1,199
  • 4
  • 14
  • 15

1 Answers1

2

It is normal for the new document vectors to contain negative values. The new dimensions correspond to concepts (though incomprehensible) in the lower dimensional space and a negative value means that the corresponding document is not related with that concept.

What do you mean by "results produced by cosine similarity make no sense" Cosine similarity should work fine. You can also try Pearson correlation (centered version of cosine).

Sanyo Mn
  • 799
  • 6
  • 15
  • What I mean is that if we have the following vectors: $(2, -4)$ and $(6,3)$, cosine similarity will be 0, indicating no relations between them. There is one more question, just to make sure - if I have 500 documents and 6000 terms, and the rank of term-document matrix equals 500, then the matrix $D$ in $TSD^{\top}$ will have size $500 \times 500$. Is that correct? Then, can I work with document vectors of size 500, or should I perform dimensionality reduction anyway? – user1315305 Aug 23 '13 at 22:16
  • And also, in similarity computations, do I use document vectors taken directly from matrix $D$, or should I use $SD^{\top}$ (the second solution is given e.g. on Wikipedia). – user1315305 Aug 23 '13 at 22:25
  • When the vectors have small sizes like 1 or 2 then the results of cosine or Pearson usually don't make sense and sometimes can be even counter-intuitive. So I suggest that you should use these measures when the vectors have larger sizes. As for your second question, there is no simple rule which says how much reduction is ok, you need to have some performance metric to find out how much reduction is best. For example if your task is document classification and if you have a dataset then you should do standard cross-validation experiments to find out the best amount of reduction. – Sanyo Mn Aug 23 '13 at 22:37
  • Since $S$ is a diagonal matrix multiplying a matrix by $S$ will only change the magnitudes of the vectors in the matrix, so cosine similarity between vectors in the matrix will not change. That is, you can use both of them. – Sanyo Mn Aug 23 '13 at 22:51
  • I have one more question - how can I convert cosine similarity into distance value when using LSA? In traditional vector space model, it was $1 - cosine$, but I guess it won't work correctly in this case? – user1315305 Aug 24 '13 at 13:11
  • If you want to make it a distance measure, first calculate the cosine of the angle of the vectors $A$ and $B$ of lengh $n$ by $$ cos \theta = \frac{ \sum \limits_{i=1}^{n}A_{i} B_{i} }{ \sqrt{\sum \limits_{i=1}^{n} A_{i}^2} \cdot \sqrt{\sum \limits_{i=1}^{n} B_{i}^2} } $$ then use the arc-cosine function to find the angle between 0-180 degree range. – Sanyo Mn Aug 24 '13 at 15:15