Quick Start Chapter 2: Hyperparameter Search

For the TensorFlow MNIST example discussed on the previous page, a random hyperparameter search can be started with the command:

pedl e create random.yaml model_def.py

To run our proprietary search algorithm on the same example, use adaptive.yaml instead of random.yaml.

The type of hyperparameter search is specified within the experiment configuration file under the searcher fields. For further documentation see the hyperparameter search page. We'll give a brief overview here.

Experiment Configuration File

The experiment configuration file specifies metadata for the experiment, including how the experiment should be deployed on a cluster and how data storage should be handled. For further documentation see the experiment configuration page.

The main fields include:

  • description: A human-readable string describing the experiment.
  • data: Where to find the dataset and how to process it.
  • checkpoint_storage: For saving the progress of an experiment.
  • hyperparameters: Defines the hyperparameters of the model.
  • searcher: Specifies the hyperparameter search algorithm and searcher-related configurations.
  • resources: Specifies constraints on the cluster resources that this experiment is allowed to use.

Search Algorithms

A hyperparameter search algorithm works on a model, defined ranges of the model's hyperparameters, and a dataset; its goal is to find the set of hyperparameter values within the hyperparameter space that results in the best validation performance for the model. We call this algorithm the searcher.

Searchers may make many decisions on the experiment procedure, including which trials and how many trials to spawn, when to calculate validation metrics on the current state of trials, whether to continue training certain trials, and which trials should be terminated. The searcher collects validation data on how well each trial is performing.

The searchers provided by PEDL are:

The const.yaml file uses the single searcher mode, which runs a single trial with fixed hyperparameters for a user-specified number of steps (so it is quite a trivial searcher). The random searcher mode trains a specified number of trials for a specified number of steps; the trials are randomly generated by sampling from the configured hyperparameter space. The grid searcher mode runs a specified "grid" of hyperparameter values for a specified number of steps. The adaptive and adaptive_simple modes implement proprietary search algorithms based on Asynchronous Successive Halving Search and Hyperband.

Hyperparameter versus Searcher Fields

The hyperparameters field defines the space of hyperparameters that will be searched over. The hyperparameter space to use for a given experiment is user-defined; importantly, hyperparameters can control aspects of the network architecture (dropout, number of layers, etc.), the training procedure (batch size, learning rate, etc.), and data preprocessing/augmentation.

On the other hand, the searcher field configures how the hyperparameter space will be explored. The name subfield defines the search algorithm to use (e.g., random, grid, etc.). Each search algorithm has additional configuration settings that should be specified as subfields of searcher.

The [mnist_tf](examples/mnist_tf.tgz) example includes .yaml configuration files for each of the hyperparameter search algorithms supported by PEDL. A side-by-side comparison of all .yaml files for mnist_tf may be helpful.