So in the code below, which is pretty standard LSTM training for the IMDB dataset, I have run extensive experiments where I changed the drop-out value from 0.5 all the way up to 1, and the accuracy on the test set is ALWAYS ~78%, never deviating by that very much. I even tried extremely low drop-out values, such as 0.2 and 0.1, and got the exact same result. Does anyone know why the accuracy of the trained model seems impervious to the drop-out rate?
Going along with the same code, I then happened to tweak a few parameters: changed the learning rate from 0.001 to 0.005, and the batch size from 32 to 25, and NOW the drop-out values do have a large impact on the accuracy of the trained model. Again, why is that?
from __future__ import division,
print_function, absolute_import
import tflearn
from tflearn.data_utils import to_categorical, pad_sequences
from tflearn.datasets import imdb
# IMDB Dataset loading
train, valid, test = imdb.load_data(
path='imdb.pkl', n_words=10000, valid_portion=0.1)
trainX, trainY = train
testX, testY = test
validX, validY = valid
# Data preprocessing
# Sequence padding
trainX = pad_sequences(trainX, maxlen=100, value=0.)
validX = pad_sequences(validX, maxlen=100, value=0.)
testX = pad_sequences(testX, maxlen=100, value=0.)
# Converting labels to binary vectors
trainY = to_categorical(trainY, nb_classes=2)
validY = to_categorical(validY, nb_classes=2)
testY = to_categorical(testY, nb_classes=2)
# Network building
net = tflearn.input_data([None, 100])
net = tflearn.embedding(net, input_dim=10000, output_dim=128)
net = tflearn.lstm(net, 128, dropout=0.1)
net = tflearn.fully_connected(net, 2, activation='softmax')
net = tflearn.regression(
net, optimizer='adam', learning_rate=0.001,
loss='categorical_crossentropy')
# Training
model = tflearn.DNN(net, tensorboard_verbose=0)
model.fit(trainX, trainY, n_epoch = 3,
validation_set=(validX, validY), show_metric=True,
batch_size=32)
score = model.evaluate(testX, testY)
print('Test accuracy: %0.4f%%' % (score[0] * 100))
