4

If I have a LSTM with below parameters, how can I calculate the total number of wights ?

Input 39
Output 34
Hidden Layers = 3
Cells in each layer = 1024

I saw the Grave Presentation. In this file, in page 7 there is a sample of LSTM with these parameters:

Input = 205
Output = 205
Hidden Layers = 5
Cells in each layer = 700

unfortunately, I can't understand why this network has 21M weights?

Thank you.

2 Answers2

6

Each cell in the LSTM has four components: the cell weights, the input gate, the forget gate, and the output gate. Each component has weights associated with all of its input from the previous layer, plus input from the previous time step. So if there are $n_i$ cells in an LSTM layer, and $n_{i-1}$ in the earlier layer, there will be $n_{i-1} + n_i$ inputs to each component of the cell. Since there are four components, that means there are $4(n_{i-1} + n_i)$ weights associated with each cell. And since we have $n_i$ cells, that means there are $4 n_i (n_{i-1} + n_i)$ weights associated with that layer.

In your first example, we have $n_0 = 39$, $n_1 = n_2 = n_3 = 1024$, and $n_4 = 34$. So the overall number of weights is about 21M.

In the second example we have $n_0 = 205$, $n_1 = ... = n_5 = 700$, and $n_6 = 205$. So the total number of weights is about 19M.

J. O'Brien Antognini
  • 1,774
  • 1
  • 10
  • 7
  • 1
    Great answer! For completion, we might want to add the number of bias vectors which are `4n_i` per layer. – wael34218 Jun 07 '19 at 07:16
0

I think this may be my answer.

If LSTM don't use recurrent projection layer and non-recurrent projection layer, use this equivalent.

(nc*nc*4*di)+(ni*nc*4*di)+(nc*no)+(nc*3*di)

nc = number of LSTM cells
ni = number of input
no = numbre of output
di = number of layers

if LSTM use projection layer, number of weights obtain from this:

(nc*nr*4*di)+(ni*nc*4*di)+((nr+np)*no)+(nc*nr*di)+(nc*3*di)+(nr*nc*4*(di-1))

nr = number of recurrent projection layer
np = number of non-recurrent projection layer

For my exmaple : I use recurrent layer too. Recurrent projection layer = 256 Non-recurrent projection layer = 256

(1024*256*4*3)+(39*1024*4*3)+((256+256)*34)+(1024*256*3)+(1024*3*3)+(256*1025*4*(2)) = 6535168 = 6.5M

I used this reference: LONG SHORT-TERM MEMORY BASED RECURRENT NEURAL NETWORK ARCHITECTURES FOR LARGE VOCABULARY SPEECH RECOGNITION