Custom Environments

PEDL launches trials of experiments, called "trial runners", and PEDL commands in customizable Docker containers. The configuration of the container is referred to as the environment.

Default Environment

In the current version of PEDL, the trial runners and commands are executed in containers with the following default settings:

  • Ubuntu 16.04
  • CUDA 10.0
  • Python 3.6.9
  • TensorFlow 1.14.0
  • PyTorch 1.2.0
  • Keras 2.2.4

PEDL will automatically select GPU-specific versions of each library when running on agents with GPUs. The trial environment can be further customized by running an arbitrary list of "runtime" commands and installing additional Python packages, as described below.

In addition to the above settings, all trial runner containers are launched with additional PEDL-specific harness code that orchestrates model training and evaluation in the container. Trial runner containers are also loaded with the experiment's model definition and values of the hyperparameters for the current trial.

Custom Environments

Customizing environments can be done by specifying environment variables for a task container or by specifying external packages to be installed.

Customizing environments can be done in two ways: either by passing files to specify conda or pip environments on the command line, or by setting fields under the environment key in the experiment configuration reference.

Environment Variables

For both trial runners and commands, PEDL allows users to configure the environment variables inside the container through the environment -> environment_variables configuration field of the experiment config. The format is a list of strings in the format NAME=VALUE:

environment:
  environment_variables:
    - A=hello world
    - B=$A
    - C=${B}
    # `A`, `B`, and `C` will each have the value `hello_world` in the container.

Variables are set sequentially, which can affect variables that depend on expansion of other variables.

External Packages

If a workload has dependencies that are not part of the default environment, additional dependencies can either be added via the environment field of the experiment config, or by passing additional config files to the pedl command line utility. The external dependencies may be pip dependencies, conda dependencies, or apt dependencies.

pip Packages

pip packages may be added to the environment -> runtime_packages field. Before launching the trial or command, the PEDL agent will build and cache a new container with the requested packges installed and commands executed.

Note

If you are running in an environment where agents do not have direct Internet access, PEDL can be configured to automatically use an HTTP proxy -- see the documentation.

Here is an example experiment configuration that uses the runtime_packages option:

environment:
  runtime_packages:
    - pandas
    - numpy

This instructs PEDL to install these packages (via pip) into the container before running any workloads.

Packages can additionally be restricted with version specifiers or installed from custom external sources using the standard pip install syntax:

environment:
  runtime_packages:
    - pandas==0.20.3 # Install this exact version
    - pandas>=0.20.0 # Install a version greater than or equal to this version
    - pandas!=0.20.1 # Exclude this version from installation
    - git+https://github.com/pandas-dev/pandas # Installing from GitHub

If there is a large number of pip dependencies, a requirements.txt of the same format as the output of pip freeze can be passed using the --requirement or -r option when launching a task. Note that the output of pip freeze will contain many packages, and users should not attempt to overwrite the primary packages for the PEDL environment, which are listed in Default Environment. Overwriting these packages may cause PEDL to behave in unexpected ways. For example, when porting an environment based on tensorflow==1.13.1 to PEDL, users should specify the version of tensorflow by setting environment -> tensorflow: "1.13.1" rather than overwrite the PEDL-provided version of tensorflow using pip. See the experiment configuration reference for all available versions of these primary packages.

pedl experiment create adaptive.yaml . --requirement requirements.txt
pedl notebook start -r requirements.txt

conda Packages

conda packages may be installed by passing a yaml-format conda environment of the same format as the output of conda env export to the --conda-env or -e option when launching a task. As with requirements.txt, it is possible to request packages which are in conflict with the environment required by PEDL. However, unlike pip, conda will explicitly throw an UnsatisfiableError due to the conflicting requirements. Users encountering this error should either relax the requirements of their conda environment to be compatible with the PEDL environment or alter the versions in the PEDL environment using the environment key in the experiment configuration reference.

pedl experiment create adaptive.yaml . --conda-env conda.yaml
pedl notebook start -e conda.yaml

apt Packages

apt dependencies may be added to the environment -> runtime_commands field. runtime_commands should be used for packages which neither pip nor conda can accommodate (e.g., installing native libraries):

environment:
  runtime_commands:
    - echo "Installing python3-pandas via apt-get"
    - apt-get install python3-pandas

The contents of runtime_commands will be executed in the order provided and before any runtime_packages are installed.

CPU and GPU Environments

runtime_commands and runtime_packages support installing different dependencies for GPU vs. CPU environments. For example:

environment:
  runtime_packages:
    cpu:
      - tensorflow
    gpu:
      - tensorflow-gpu

Local Packages

For trial runners, PEDL also supports installing dependencies from a user's development environment, as long as the local package is a Python source distribution file or a Python wheel file that supports installation into a linux_x86_64 and py3 environment. To use a local package, specify the --package flag when creating the experiment:

$ pedl experiment create config.yaml <model definition directory> \
    --package user-package-1.tar.gz \
    --package user-package-2.tar.gz

The command above will install user-package-1.tar.gz and user-package-2.tar.gz on all trial runners in the experiment. There is no limit on the number of packages that can be specified. Note that the total size of the model definition and all packages must be less than 96 MB.

Custom Docker Images

Warning

This is currently an experimental feature.

PEDL supports running trials and executing commands based on user-provided Docker images. The image in custom_image should be accessible to all agent nodes via docker pull. If a private image is used, Docker Registry credentials must be specified in the registry_auth section in the experiment configuration.

When provided with a custom image, PEDL will build a runtime image that imports the custom_image with a FROM instruction, injects the Python PEDL source code with COPY instructions, and installs Python requirement libraries with pip. The maintainer of the custom image is responsible for installing the following PEDL dependencies:

  • Git >= 1.5.0
  • Python 3.6.9 installed as python3.6
  • Pip installed as pip
  • CUDA 10.0
  • CuDNN 7.4
  • Framework-specific libraries, depending on the interface used:
    • TensorFlow 1.12, 1.13, or 1.14 if using TensorFlow or Keras APIs
    • PyTorch 1.1.0 or 1.2.0 if using the PyTorch API

If you would like to use versions of libraries different from those specified above in a custom base image, please contact the Determined AI team for a consultation.