Generating Sentence Vectors from Word2Vec

Question

I know that I can use doc2Wec and other resources to get sentence vectors. But I am very curious to generate sentence vectors using Word2Vec.

I read lot of materials and found that Averaging the embeddings is the baseline architecture but it is not clear on what axis we have to perform the averaging. Even though 'Averaging Word Vectors' clearly mentions that I have to average out the embeddings of every vector i feel this will have disadvantage that my sentence vectors will be of unequal lengths. Consider below example.

Here every word is represented in 5 dimensions. Now lets try to generate sentence vector for the sentence

'how are you' and 'fine'

My 'how are you' sentence will be of shape 1 row and 3 columns(3 words in this sentence)
'fine' sentnece will be of shape 1 row and 1 column..

Now we can see that both the sentences are of unequal length.. So I will have to append 0s to shorter sentence to match the length. This will be adding redundancy to my data. So what are the best ways to get the vectors of equal length without using doc2vec and avoiding zero padding.

Answer 1

The standard fast solution is really averaging the embeddings. The problem of averaging is that function words that appear in most sentences cause that the longer the sentence is, the more it converges to non-informative average. Therefore removing stop words is necessary in this case to get reasonable performance.

Methods based on spectral matrix decomposition might also be an interesting alternative.

Nevertheless, I would say that nowadays, representing a sentence using static word embeddings is a poor man's solution that comes into play when speed or memory requirements really matter.

When you represent a sentence using static word embeddings, the main weakness is that embeddings are not informed about the context in which the words are used. This is the reason why contextual word representations such as ELMo or BERT has been developed and why they are so successful. They provide an answer that you probably don't want to hear: build a large recurrent or Transformer network and pre-train it as a language model. This is currently the best way how to represent a sentence when you start with word embeddings.