Shortcuts

Install Determined on Kubernetes

This document describes how to install Determined on Kubernetes. The installation is performed using the Determined Helm Chart. For general information about using Determined with Kubernetes, refer the Determined on Kubernetes guide.

Prerequisites

Before installing Determined on a Kubernetes cluster, please ensure that the following prerequisites are satisfied:

  • The Kubernetes cluster should be running Kubernetes >= 1.15, with GPU support enabled.

  • You should have access to the cluster via kubectl.

  • Helm should be installed.

You should also download a copy of the Determined Helm Chart and extract it on your local machine.

If you do not yet have a Kubernetes cluster deployed and you want to use Determined in a public cloud environment, we recommend using a managed Kubernetes offering such as Google Kubernetes Engine (GKE) on GCP or Elastic Kubernetes Service (EKS) on AWS. For more info on configuring GKE for use with Determined, refer to our Instructions for setting up a GKE cluster.

What Gets Installed

When the Determined Helm chart is installed, the following entities will be created:

  1. Deployment of the Determined master.

  2. ConfigMap containing configurations for the Determined master.

  3. LoadBalancer service to make the Determined master accessible.

  4. ServiceAcccount which will be used by the Determined master.

  5. Deployment of a Postgres database. Later in this guide, we describe how an external database can be used instead.

  6. PersistentVolumeClaim for the Postgres database. Omitted if using an external database.

  7. Service to allow the Determined master to communicate with the Postgres database. Omitted if using an external database.

Configuration

When installing Determined using Helm, you should first configure some aspects of the Determined deployment by editing the values.yaml and Chart.yaml files in the Helm chart.

Version Configuration

To configure which version of Determined will be installed by the Helm chart, users should modify appVersion in Chart.yaml. Users can specify a release version (e.g., 0.13.0) or specify any commit hash from the upstream Determined repo (e.g., b13461ed06f2fad339e179af8028d4575db71a81). Users are strongly encouraged to use a released version.

Number of GPUs Per Node

Users are required to specify the number of GPUs on each node (for GPU-enabled nodes only). This is done by setting maxSlotsPerPod in values.yaml. Determined uses this information when scheduling multi-GPU tasks. Each multi-GPU (distributed training) task will be scheduled as a set of slotsPerTask / maxSlotsPerPod separate pods, with each pod assigned up to maxSlotsPerPod GPUs. Distributed tasks with sizes that are not divisible by maxSlotsPerPod are never scheduled. If you have a cluster of different size nodes set maxSlotsPerPod to the smallest common denominator. For example, if you have nodes with 4 GPUs and other nodes with 8 GPUs, set maxSlotsPerPod to 4 so that all distributed experiments will launch with 4 GPUs per pod (e.g., on nodes with 8 GPUs, two such pods would be launched).

Checkpoint Storage

Checkpoints and TensorBoards events can be configured to be stored in shared_fs, AWS S3, or GCS. By default, checkpoints and TensorBoard events are stored using shared_fs, which creates a hostPath Volume and saves to the host file system. This configuration is intended for initial testing only; users are strongly discouraged from using shared_fs for actual deployments of Determined on Kubernetes, because most Kubernetes cluster nodes do not have a shared file system.

Instead of using shared_fs, users should configure either AWS S3 or GCS:

  • AWS S3: To configure Determined to use AWS S3 for checkpoint and TensorBoard storage, users need to set checkpointStorage.type in values.yaml to s3 and set checkpointStorage.bucket to the name of the bucket. The pods launched by the Determined master must have read, write, and delete access to the bucket. To enable this users may optionally configure checkpointStorage.accessKey and checkpointStorage.secretKey. Users may also optionally configure checkpointStorage.endpointUrl which specifies the endpoint to use for S3 clones (e.g., http://<minio-endpoint>:<minio-port|default=9000>).

  • GCS: To configure Determined to use Google Cloud Storage for checkpoints and TensorBoard data, users need to set checkpointStorage.type in values.yaml to gcs and set checkpointStorage.bucket to the name of the bucket. The pods launched by the Determined master must have read, write, and delete access to the bucket. For example, when launching their GKE nodes users need to specify --scopes=storage-full to configure proper GCS access.

Default Pod Specs (Optional)

As described in the Determined on Kubernetes guide, when tasks (e.g., experiments, notebooks) are started in a Determined cluster running on Kubernetes, the Determined master launches pods to execute these tasks. The Determined helm chart makes it possible to set default pod specs for all CPU and GPU tasks. The defaults can be defined in values.yaml under taskContainerDefaults.cpuPodSpec and taskContainerDefaults.gpuPodSpec. For examples of how to do this and a description of permissible fields please see the specifying custom pod specs guide.

Database (Optional)

By default, the Helm chart will deploy an instance of Postgres on the same Kubernetes cluster where Determined itself is deployed. If this is undesirable, users can configure the Helm chart to use an external Postgres database by setting db.hostAddress to the IP address of their database. If db.hostAddress is configured, the Determined Helm chart will not deploy a database.

Installing Determined

Once finished making configuration changes in values.yaml and Chart.yaml, Determined is ready to be installed. To install Determined run:

helm install <name for your deployment> --wait determined-helm-chart

determined-helm-chart is a relative path to where the Determined Helm Chart is located. Helm will install Determined within the default namespace. If you wish to install Determined into a non-default namespace, add -n <namespace name> to the command shown above.

Once the installation has completed, instructions will be displayed for discovering the IP address assigned to the Determined master. The IP address can also be discovered by running kubectl get services.

Upgrading Determined

To upgrade Determined or to change a configuration setting, first make the appropriate changes in values.yaml and Chart.yaml, and then run:

helm upgrade <name for your deployment> --wait determined-helm-chart

Before upgrading Determined, consider pausing all active experiments. Any experiments that are active when the Determined master restarts will resume training after the upgrade, but will be rolled back to their most recent checkpoint.

Uninstalling Determined

To uninstall Determined run:

# Please note that if the Postgres Database was deployed by Determined, it will
# be deleted by this command, permanently removing all records of your experiments.
helm delete <name for your deployment>

# If there were any active tasks when uninstalling, this command will
# delete all of the leftover Kubernetes resources. It is recommended to
# pause all experiments prior to upgrading or uninstalling Determined.
kubectl get pods --no-headers=true -l=determined | awk '{print $1}' | xargs kubectl delete pod