Shortcuts

Specifying Custom Pod Specs

As described in the Determined on Kubernetes guide, when tasks (e.g., experiments, notebooks) are started in a Determined cluster running on Kubernetes, the Determined master launches pods to execute these tasks. In this guide, we focus on how users can customize these pods by providing their own pod specs. Common use cases include but are not limited to: assigning pods to specific nodes, specifying additional volume mounts, attaching permissions. Please note that configuring pod specs is not required to use Determined on Kubernetes.

In this topic guide, we will cover:

  1. The different ways to configure custom pod specs.

  2. Supported pod spec fields.

  3. How to configuring default pod specs.

  4. How to configuring per-task pods specs.

Ways to Configure Pod Specs

Determined provides two ways to configure pod specs. When Determined is installed, the system administrator can configure pod specs that are used by default for all GPU and CPU tasks. In addition, users can specify a custom pod spec for individual tasks (e.g., for an experiment by specifying environment.pod_spec in the experiment configuration). If a custom pod spec is specified for a task, it overrides the default pod spec (if any).

Supported Pod Spec Fields

This section describes which fields users can and cannot configure when specifying custom pod specs.

Determined does not currently support configuring:

  • Pod Name - Determined automatically assigns a name for every pod that is created.

  • Pod Namespace - Determined launches all tasks in the Namespace in which the Determined master is running.

  • Host Networking - This must be configured via the Master Configuration.

  • Init Containers - Determined does not currently support init containers.

Additionally, users are allowed to specify the spec for exactly one container within the pod. The specs of this container will be applied to the container that executes the task launched by Determined. For the container spec, Determined does not currently support configuring:

  • Container Name - Determined automatically assigns this.

  • Container Image - Please see Container Images for how to specify custom docker images.

  • Container Command.

  • Container Args.

  • Container WorkingDir - Determined sets the working directory to be /run/determined/workdir.

  • Container Ports.

  • Container EnvFrom.

  • Container Env - Please see Environment Variables for how to set environment variables.

  • Container LivelinessProbe.

  • Container ReadinessProbe.

  • Container StartupProbe.

  • Container Lifecycle.

  • Container TerminationMessagePath.

  • Container TerminationMessagePolicy.

  • Container ImagePullPolicy - Please see force_pull_image for how to configure. Defaults to PullIfNotPresent.

  • Container SecurityContext.

Configuring Default Pod Specs

Defining default pod specs must be done when installing or updating Determined. The default pod specs are configured in values.yaml of the Determined helm chart under taskContainerDefaults.cpuPodSpec and taskContainerDefaults.gpuPodSpec. The gpuPodSpec is applied to all tasks that use GPUs (e.g., experiments, notebooks). cpuPodSpec is applied to all tasks that only use CPUs (e.g, TensorBoards, CPU-only notebooks).

Example of configuring default pod specs:

taskContainerDefaults:
  cpuPodSpec:
    metadata:
      labels:
        customLabel: cpu-label
    spec:
      containers:
        volumeMounts:
        - mountPath: /example-data
          name: example-volume
      volumes:
        - name: example-volume
          hostPath:
            path: /data
  gpuPodSpec:
    metadata:
      labels:
        customLabel: gpu-label
    spec:
      containers:
        volumeMounts:
        - mountPath: /example-data
          name: example-volume
      volumes:
        - name: example-volume
          hostPath:
            path: /data

Configuring Per-Task Pod Specs

In addition to default pod specs, it is also possible to configure custom pod specs for individual tasks. When defining a custom pod spec for a task, it will override the default pod spec if one is defined. Pod specs for individual tasks can be configured under the environment field in the experiment config.

Example of configuring a pod spec for an individual task:

environment:
  pod_spec:
    metadata:
      labels:
        customLabel: task-specific-label