validation accuracy, recall and precision remains constant after 30th epoch

Question

I am using TensorFlow model EfficientNetB0 for transfer learning, but after a number of epochs the validation accuracy, -precision, and -recall remains constant. Is this something I should be worried about? Also, I have 158 test files, but when I count up the values in the confusion matrix I only get 144.

Here is the code and output:

import os
import tensorflow as tf
import matplotlib.pyplot as plt
from tensorflow.keras import layers
from google.colab import drive
from keras.preprocessing.image import ImageDataGenerator
from sklearn.metrics import accuracy_score, f1_score, precision_score, confusion_matrix


img_size = (224,224)
batch = 16

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    train_dir,
    validation_split = 0.25,
    subset = 'training',
    seed=123,
    image_size= img_size,
    batch_size = batch,
    label_mode = 'binary'
)

val_ds = tf.keras.preprocessing.image_dataset_from_directory(
    train_dir,
    validation_split = 0.25,
    subset = 'validation',
    seed=123,
    image_size= img_size,
    batch_size = batch,
    label_mode ='binary'
)

test_ds = tf.keras.preprocessing.image_dataset_from_directory(
    test_dir,
    #seed=123,
    image_size= img_size,
    batch_size = batch,
    #label_mode = None
)

class_names = test_ds.class_names

AUTOTUNE = tf.data.AUTOTUNE

train_ds = train_ds.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)
test_ds = test_ds.prefetch(buffer_size=AUTOTUNE)


preprocess_input = tf.keras.applications.efficientnet.preprocess_input

data_augmentation = tf.keras.Sequential([
  tf.keras.layers.experimental.preprocessing.RandomFlip('horizontal'),
  tf.keras.layers.experimental.preprocessing.RandomFlip('vertical'),
  tf.keras.layers.experimental.preprocessing.RandomRotation(0.2)
])

base_model = tf.keras.applications.EfficientNetB0(input_shape = (224, 224, 3),
                                                    include_top = False,
                                                    weights = 'imagenet')

base_model.trainable = False
global_average_layer = tf.keras.layers.GlobalAveragePooling2D()
prediction_layer = tf.keras.layers.Dense(1)

inputs = tf.keras.Input(shape=(224,224, 3))
x = data_augmentation(inputs)
x = preprocess_input(x)
x = base_model(x, training=False)
x = global_average_layer(x)
x = tf.keras.layers.Dropout(0.2)(x)
x = prediction_layer(x)
outputs = tf.nn.sigmoid(x)
model = tf.keras.Model(inputs, outputs)

base_learning_rate = 0.0001
model.compile(optimizer=tf.keras.optimizers.Adam(lr=base_learning_rate),
              loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),
              metrics=['accuracy','Recall', 'Precision', 'FalsePositives', 'TruePositives', 'FalseNegatives'])

epochs=100
history = model.fit(
  train_ds,
  validation_data=val_ds,
  epochs=epochs
)

loss, accuracy, recall, precision, fp, tp, fn = model.evaluate(test_ds)
print('Test accuracy :', accuracy)
print('Test recall :', recall)
print('Test precision :', precision)
print('False Positive :', fp)
print('True Positive :', tp)
print('False Negative :', fn)
tn = 144-fp-tp-fn
print('True Negative :', tn)

Output:

score 0 · Answer 1 · answered Apr 29 '21 at 12:36

As long as you have real-world data, there will always be remaining variation that your models cannot account for. There may be two cases with exactly the same predictor settings, but one is class A, the other class B.

Therefore, your KPIs will not asymptotically approach 1, no matter how long you train. Instead, they will always plateau somewhere below 1.

Also: Do not use accuracy to evaluate a classifier: Why is accuracy not the best measure for assessing classification models? Is accuracy an improper scoring rule in a binary classification setting? Classification probability threshold The same problems apply to sensitivity and specificity, and indeed to all evaluation metrics that rely on hard classifications. Instead, use probabilistic classifications, and evaluate these using proper scoring rules.

validation accuracy, recall and precision remains constant after 30th epoch

1 Answers1