Understanding LSTM units vs. cells

Question

I have been studying LSTMs for a while. I understand at a high level how everything works. However, going to implement them using Tensorflow I've noticed that BasicLSTMCell requires a number of units (i.e. num_units) parameter.

From this very thorough explanation of LSTMs, I've gathered that a single LSTM unit is one of the following

which is actually a GRU unit.

I assume that parameter num_units of the BasicLSTMCell is referring to how many of these we want to hook up to each other in a layer.

That leaves the question - what is a "cell" in this context? Is a "cell" equivalent to a layer in a normal feed-forward neural network?

Answer 1

The terminology is unfortunately inconsistent. num_units in TensorFlow is the number of hidden states, i.e. the dimension of $h_t$ in the equations you gave.

Also, from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/api_docs/python/functions_and_classes/shard9/tf.nn.rnn_cell.RNNCell.md :

The definition of cell in this package differs from the definition used in the literature. In the literature, cell refers to an object with a single scalar output. The definition in this package refers to a horizontal array of such units.

"LSTM layer" is probably more explicit, example:

def lstm_layer(tparams, state_below, options, prefix='lstm', mask=None):
    nsteps = state_below.shape[0]
    if state_below.ndim == 3:
        n_samples = state_below.shape[1]
    else:
        n_samples = 1

    assert mask is not None
    […]

Answer 2

Most LSTM/RNN diagrams just show the hidden cells but never the units of those cells. Hence, the confusion. Each hidden layer has hidden cells, as many as the number of time steps. And further, each hidden cell is made up of multiple hidden units, like in the diagram below. Therefore, the dimensionality of a hidden layer matrix in RNN is (number of time steps, number of hidden units).

Answer 3

Although the issue is almost the same as I answered in this answer, I'd like to illustrate this issue, which also confused me a bit today in the seq2seq model (thanks to @Franck Dernoncourt's answer), in the graph. In this simple encoder diagram:

Each $h_i$ above is the same cell in different time-step (cell either GRU or LSTM as that in your question) and the weight vectors(not bias) in the cell are of the same size of (num_units/num_hidden or state_size or output_size).

RNN is a special type of graphical model where nodes form a directed list as explained in section 4 of this paper: Supervised Neural Networks for the Classication of Structures. We can think of num_units as the number of tags in CRF(although CRF is undirected), and the matrices($W$'s in graph in the question) are all shared across all time steps like the transition matrix in CRF.

Answer 4

In keras.layers.LSTM(units, activation='tanh', ....), the units refers to the dimensionality or length of the hidden state or the length of the activation vector passed on the next LSTM cell/unit - the next LSTM cell/unit is the "green picture above with the gates etc from http://colah.github.io/posts/2015-08-Understanding-LSTMs/

The next LSTM cell/unit (i.e. the green box with gates etc from http://colah.github.io/posts/2015-08-Understanding-LSTMs/) is NOT the same as the units in keras.layers.LSTM(units, activation='tanh', ....)

The units are also sometimes called the latent dimensions. Here is a detailed explanation of the units LSTM parameter:

https://zhuanlan.zhihu.com/p/58854907

Answer 5

Quoting from TF's tutorial on RNNs:

In addition to the built-in RNN layers, the RNN API also provides cell-level APIs. Unlike RNN layers, which processes whole batches of input sequences, the RNN cell only processes a single timestep.

Answer 6

In my opinion, cell means a node such as hidden cell which is also called hidden node, for multilayer LSTM model,the number of cell can be computed by time_steps*num_layers, and the num_units is equal to time_steps