Custom Environment

Determined launches workloads using Docker containers. By default, workloads execute inside a Determined-provided container that includes common deep learning libraries and frameworks.

If your model code has additional dependencies, the easiest way to install them is to specify a startup hook. For more complex dependencies, use a custom Docker image.

If you are using Determined on Kubernetes, review the Custom Pod Specs guide.

Environment Variables

For both trial runners and commands, you can configure the environment variables inside the container using the experiment or task environment.environment_variables configuration field. The format is a list of NAME=VALUE strings. For example:

environment:
  environment_variables:
    - A=hello world
    - B=$A
    - C=${B}

Variables are set sequentially, which affect variables that depend on the expansion of other variables. In the example, names A, B, and C each have the value hello_world in the container.

Proxy variables set in this way take precedent over variables set in the agent configuration.

You can also set variables for each accelerator type, separately:

environment:
  environment_variables:
    cpu:
      - A=hello x86
    gpu:
      - A=hello nvidia
    rocm:
      - A=hello amd

Startup Hooks

If a startup-hook.sh file exists in the top level of your model definition directory, this file is automatically run with every Docker container startup. This occurs before any Python interpreters are launched or deep learning operations are performed. The startup hook can be used to customize the container environment, install additional dependencies, and download data sets among other shell script commands.

Startup hooks are not cached and run before the start of every workload so expensive or long-running operations in a startup hook can result in poor performance.

Example startup hook that installs the wget utility and the pandas Python package:

apt-get update && apt-get install -y wget
python3 -m pip install pandas

And, this Iris example contains a TensorFlow Keras model that uses a startup hook to install an additional Python dependency.

Container Images

Officially supported, default Docker images are provided to launch containers for experiments, commands, and other workflows.

All trial runner containers are launched with additional Determined-specific harness code, which orchestrates model training and evaluation in the container. Trial runner containers are also loaded with the experiment model definition and hyperparameter values for the current trial.

GPU-specific versions of each library are automatically selected when running on agents with GPUs.

Default Images

Environment

File Name

CPUs

determinedai/environments:py-3.8-pytorch-1.10-lightning-1.5-tf-2.8-cpu-0.17.15

Nvidia GPUs

determinedai/environments:cuda-11.3-pytorch-1.10-lightning-1.5-tf-2.8-gpu-0.17.15

AMD GPUs

determinedai/environments:rocm-4.2-pytorch-1.9-tf-2.5-rocm-0.17.15

Custom Images

While the official images contain all the dependencies needed for basic deep learning workloads, many workloads have additional dependencies. If the extra dependencies are quick to install, you might consider using a startup hook. Where installing dependencies using startup-hook.sh takes too long, it is recommended that you build your own Docker image and publish to a Docker registry, such as Docker Hub.

Warning

Do NOT install TensorFlow, PyTorch, Horovod, or Apex packages, which conflict with Determined-installed packages.

It is recommended that custom images use one of the official Determined images as a base image, using the FROM instruction.

Example Dockerfile that installs custom conda-, pip-, and apt-based dependencies:

# Determined Image
FROM determinedai/environments:cuda-11.3-pytorch-1.10-lightning-1.5-tf-2.8-gpu-0.17.15

# Custom Configuration
RUN apt-get update && \
   DEBIAN_FRONTEND="noninteractive" apt-get -y install tzdata && \
   apt-get install -y unzip python-opencv graphviz
COPY environment.yml /tmp/environment.yml
COPY pip_requirements.txt /tmp/pip_requirements.txt
RUN conda env update --name base --file /tmp/environment.yml
RUN conda clean --all --force-pkgs-dirs --yes
RUN eval "$(conda shell.bash hook)" && \
   conda activate base && \
   pip install --requirement /tmp/pip_requirements.txt

Assuming that this image is published to a public repository on Docker Hub, use the following declaration format to configure an experiment, command, or notebook:

environment:
  image: "my-user-name/my-repo-name:my-tag"

where my-user-name is your Docker Hub user, my-repo-name is the name of the Docker Hub repository, and my-tag is the image tag to use, such as latest.

If you publish your image to a private Docker Hub repository, you can specify the credentials needed to access the repository:

environment:
  image: "my-user-name/my-repo-name:my-tag"
  registry_auth:
    username: my-user-name
    password: my-password

If you publish the image to a private Docker Registry, specify the registry path as part of the image field:

environment:
  image: "myregistry.local:5000/my-user-name/my-repo-name:my-tag"

Images are fetched using HTTPS by default. An HTTPS proxy can be configured using the https_proxy field in the agent configuration.

The custom image and credentials can be set as the defaults for all tasks launched in Determined, using the image and registry_auth fields in the Master Configuration. Make sure to restart the master for this to take effect.

Virtual Environments

Model developers commonly use virtual environments. The following example configures virtual environments using custom images:

# Determined Image
FROM determinedai/environments:py-3.8-pytorch-1.10-lightning-1.5-tf-2.8-cpu-0.17.15

# Create a virtual environment
RUN conda create -n myenv python=3.6
RUN eval "$(conda shell.bash hook)" && \
   conda activate myenv && \
   pip install scikit-learn

# Set the default virtual environment
RUN echo 'eval "$(conda shell.bash hook)" && conda activate myenv' >> ~/.bashrc

To ensure that a virtual environment is activated every time a new interactive terminal session is created, in JupyterLab or using Determined Shell, update ~/.bashrc with the scripts to activate the virtual environment you want.

This example switches to a virtual environment using a startup hook:

# Switch to the desired virtual environment
eval "$(conda shell.bash hook)"
conda activate myenv

# Do that for every new interactive terminal session
echo 'eval "$(conda shell.bash hook)" && conda activate myenv' >> ~/.bashrc