How does Keras generate an LSTM layer. What's the dimensionality?

Question

In Keras I can define the input shape of an LSTM (and GRU) layers by defining the number of training data sets inside my batch (batch_size), the number of time steps and the number of features.

So I could configure an LSTM or a GRU like that: batch_input_shape=(BATCH_SIZE,TIME_STEPS,FEATURES)

I would like to understand what that means in detail.

When my training data looks like that:

trainingData = [
   [[0.66804451], [0.7008307], ...,[0.89589089]], //one input with 30 time steps
   [[0.89773387], [0.89903724], ..., [0.71692085], //another input with 30 time steps ...
   ...
]

trainingData          //contains my batches
trainingData[0]       //contains one training set
trainingData[1]       //contains another training set
trainingsData[0][0]   //contains the first timestep in the first data set.

and my models looks like that:

model = K.models.Sequential()
model.add(K.layers.LSTM(UNITS,
                    return_sequences=False,
                    input_shape=(TIMESTEPS_INPUT, 1),
                    batch_size=1000,
                    name="GRU_2"))

Now, the first number when calling "LSTM(...)" UNITS means that each LSTM cell has a history of 5 values, so there are 4 more LSTM-cells behind a single LSTM cell. When we unroll the LSTM we can see 5 LSTM-cells, right?

The question now is, what TIME_STEPS(=30) means. I think there are two possibilities:

First: It could mean, that the LSTM-layer gets 2 dimensional with UNITSxTIME_STEPS. Which would mean, that I have 30x5 LSTM cells on the first layer?

Second: Maybe TIME_STEPS just defines the number of inputs and input_weights on one LSTM-cell-column (with 5 LSTM cells). An LSTM layer would always just contain one "column" of LSTM cells (number of cells defined by #UNITS), which can be unrolled. There would be just 5 LSTM cells connected to each other, when I set UNITs to 5. Each cells would get 30 inputs?

If it is the seconds possibility. Is there a way to create more "LSTM-cell-columns" in one layer?

if you thought my response was helpful could you mark it as the chosen answer? =) — Luciano Viola, Sep 19 '18 at 17:18

Luciano Viola · Answer 1 · 2017-11-10T13:47:47.870

Now, the first number when calling "LSTM(...)" UNITS means that each LSTM cell has a history of 5 values, so there are 4 more LSTM-cells behind a single LSTM cell. When we unroll the LSTM we can see 5 LSTM-cells, right?

Not really. "Units" means how many neurons (or cells) your network will have. This network will then be copied (unfolded) t number of times, where t is the size of your sequence (timesteps).

The question now is, what TIME_STEPS(=30) means?

This is the number that will define how many times your network will "unfold" through time.

Consider the following image:

In your case:

Inside each LSTM (blue) box you will have 5 neurons.

The image shows the network unfolding 3 times. In your case, it would unfold 30 times (it will repeat this whole process for every sample in your data).

How does Keras generate an LSTM layer. What's the dimensionality?

1 Answers1