0

I know there is same question here. But I couldn't get a satisfactory answer and I want to ask something different.

In Skip-gram model, 2nd answer of above post and Word2Vec Explained say the same thing that output C multinomial distributions are generated by same hidden -> output weight matrix.

1) By same N by V dimension matrix, how can different output vectors be generated?

What I figured out was, same C output vectors are generated and than cross entropy losses are calculated by C different true labels. For example, in 'the cat jumped over the table', given 'jumped' as an input, it is fed forward through the network to produce an output vector O. Then the losses are computed with O compared with 'cat', 'over' vectors, when C = 2. Then we sum up the losses and back-propagate using that summed-up loss to update the shared weight matrix.

2) Figure below is from Word2vec explained by Xin Rong. I don't think I understand the meaning of u_c,j. What is the 'net input' of j-th unit of c-th panel of output layer? Word2vec explained

Peter Kim
  • 121
  • 4

2 Answers2

0
  1. I think that neural will return C identical vectors V-dims.Then we will the actual vector correspond to these vectors.
  2. It simple is the input of set consist of C panel(each panel is a vector V-dims)
ducPham
  • 11
  • 2
-1

Yes it is a bit ambiguous ,IMHO :

(net input) : U_c,j is the input (net input) that we give to the last layer ,it is computed from the hidden and the output matrix , and then to get the final output (Y_c,j ) we apply softmax function on the input (U_c,j) .

j-th unit of c-th panel : each C panel is a vector (one-hot representation of All the vocabulary) ,so depending on the (U_c,j ) input - witch is similaire for all other C panels : U_c,j =U_j - we will have the probability that this (j-th unit) is probable or not .

Tou You
  • 101
  • 2