TensorFlow Keras CNN Tutorial

In this tutorial, we show you how to use the PEDL API to train an image classifier based on the CIFAR10 image dataset. This tutorial is inspired by the official TensorFlow Convolutional Neural Network Tutorial.

This tutorial requires the PEDL CLI. For installation procedures: Install PEDL CLI

Overview

In this tutorial, we will walk you through a PEDL implementation of the CIFAR10 image classifier. The full code is available to download at CIFAR-10 CNN or in the Code Sample section.

PEDL requires two functions to build a TensorFlow Keras model:

  1. make_data_loaders

  2. build_model

In our code example, the file main.py contains both of these functions.

Additionally, the PEDL frameworks expects two files to be provided:

  1. an entry point (__init__.py)

  2. an experiment configuration file (const.yaml)

__init__.py is our entry point. It imports make_data_loaders and the model class, which contains build_model. The experiment configuration file (*.yaml) contains information on hyperparameters as well as details on the searcher such as the metric used for optimization.

Downloading and Preparing the CIFAR10 Dataset

To download the CIFAR10 dataset we the use the dataset API from tf.keras.

from tensorflow.keras import datasets

Downloading the test and train images with labels fits in a single line of code:

(train_images, train_labels), (test_images, test_labels) = datasets.CIFAR10.load_data()
../_images/CIFAR10.png

For training, we want to rescale the image RGB values to lie between 0 and 1. We define a preprocess_data function for this purpose.

def preprocess_data(data):
    return data / 255.0

We also convert the image labels to integers via tf.keras’s utils.to_categorical feature.

def preprocess_labels(labels):
    return utils.to_categorical(labels)

Next, we need to pass this data to PEDL by defining the make_data_loaders function. The make_data_loaders takes in hyperparameters and returns two batched sequences - one for train images and one for test images.

def make_data_loaders(hparams)
    # Obtain and batch training and test data
    return train, test

In our example, we define the train and test sequence as

train = keras.data.InMemorySequence(
    data=preprocess_data(train_images),
    labels=preprocess_labels(train_labels),
    batch_size=batch_size,

test = keras.data.InMemorySequence(
    data=preprocess_data(test_images),
    labels=preprocess_labels(test_labels),
    batch_size=batch_size,
)

For distributed training we can make use of PEDL’s (optional) download_data API. For more information about PEDL’s download_data function check out PyTorch Data Downloading.

Building the Model

Each PEDL experiment expects the implementation of a trial class. This class must inherit from an appropriate base class that corresponds to the framework we would like to use.

In our case, we are using a TFKeras model and thus our trial class subclasses PEDL’s TFKerasTrial class. In the trial class, we define our build_model function.

We will use the original tf.keras libraries. Specifically, we load the layers, models and utils modules.

from tensorflow.keras import layers, models, utils

Then we define our model within the CIFARTrial class.

class CIFARTrial(TFKerasTrial):
    def build_model(self, hparams):
    # Define the model
    return model

We define the Sequential CNN model as we would in TFKeras:

model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation="relu", input_shape=(32, 32, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation="relu"))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation="relu"))

model.add(layers.Flatten())
model.add(layers.Dense(64, activation="relu"))
model.add(layers.Dense(10, activation="softmax"))

model.compile(
    optimizer="adam", loss="categorical_crossentropy", metrics=["categorical_accuracy"],
)

To view all model code in one place either download it from CIFAR-10 CNN or go to the Code Sample section.

Training the Model

Lastly, we specify our hyperparameters and the optimization metric in our experiment configuration *.yaml file.

description: CIFAR10_keras_const
hyperparameters:
  batch_size: 32
searcher:
  name: single
  metric: val_categorical_accuracy
  max_steps: 50
min_validation_period: 10

In this simple example, we only set one hyperparameter, which is batch size = 32. We will not be using any hyperparameters tuning and thus our searcher is single. We use val_categorical_accuracy as our searcher metric because CIFAR10 is a classification problem. We run the experiment for a total of 50 steps.

In PEDL, the runtime of an experiment is defined in steps instead of epochs. Each step consists of 100 batches. Thus, in our example, with batch size equal to 32 and 50 steps, we run through a total of 32*50*100 = 160k samples. Finally, min_validation_period specifies after how many steps we compute validation metrics.

Now that the experiment configuration is set up, we are ready to run our experiment with a single command:

pedl experiment create const.yaml .

Here, the first argument (const.yaml) specifies the experiment configuration file and the second argument (.) the location of the directory that contains our model definition files.

Once the experiment is started, you will see a notification like this:

Preparing files (/cifar10_cnn_tf) to send to master... 3.2KB and 4 files
Created experiment xxx
Activated experiment xxx

Evaluating the Model

Thanks to PEDL, model evaluation is done automatically for you. To access information on both training and validation performance, simply go to the webUI by entering the address of your PEDL_MASTER in your web browser.

Once you are on the PEDL landing page, you can find your experiment either via the experiment ID (xxx) or via its description.

Code Sample

Putting it all together, our main.py file looks like this.

from typing import Any, Dict, List, Tuple
from tensorflow.keras import datasets, layers, models, utils
from pedl.frameworks import keras
from pedl.frameworks.keras import TFKerasTrial

def preprocess_data(data):
    return data.astype("float32") / 255

def preprocess_labels(labels):
    return utils.to_categorical(labels)

class CIFARTrial(TFKerasTrial):
    def build_model(self, hparams):

        model = models.Sequential()
        model.add(layers.Conv2D(32, (3, 3), activation="relu", input_shape=(32, 32, 3)))
        model.add(layers.MaxPooling2D((2, 2)))
        model.add(layers.Conv2D(64, (3, 3), activation="relu"))
        model.add(layers.MaxPooling2D((2, 2)))
        model.add(layers.Conv2D(64, (3, 3), activation="relu"))

        model.add(layers.Flatten())
        model.add(layers.Dense(64, activation="relu"))
        model.add(layers.Dense(10, activation="softmax"))

        model.compile(
            optimizer="adam", loss="categorical_crossentropy", metrics=["categorical_accuracy"],
        )

        return model

def make_data_loaders(experiment_config, hparams):
    (train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()

    batch_size = hparams["batch_size"]

    train = keras.data.InMemorySequence(
        data=preprocess_data(train_images),
        labels=preprocess_labels(train_labels),
        batch_size=batch_size,
    )

    test = keras.data.InMemorySequence(
        data=preprocess_data(test_images),
        labels=preprocess_labels(test_labels),
        batch_size=batch_size,
    )
    return train, test

This code is also available at CIFAR-10 CNN.

Next Steps