The behavior of the master and agent can be controlled by setting configuration variables; this can be done using a configuration file, environment variables, or command-line options. Although values from different sources will be merged, we generally recommend sticking to a single source for each service to keep things simple.
The master and the agent both accept an optional
command-line option, which specifies the path of the configuration file
to use. Note that when running the master or agent inside a container,
you will need to make the configuration file accessible inside the
container (e.g., via a bind mount). For example, this command starts the
agent using a configuration file:
docker run \ -v `pwd`/agent-config.yaml:/etc/determined/agent-config.yaml \ determinedai/determined-agent --config-file /etc/determined/agent-config.yaml
agent-config.yaml file might contain
master_host: 127.0.0.1 master_port: 8080
to configure the address of the Determined master that the agent will attempt to connect to.
Each option in the master or agent configuration file can also be
specified as an environment variable or a command-line option. To
configure the behavior of the master or agent using environment
variables, specify an environment variable starting with
followed by the name of the configuration variable. Underscores (
should be used to indicate nested options: for example, the
scheduler.fit master configuration option can be specified via an
environment variable named
The equivalent of the agent configuration file shown above can be
specified by setting two environment variables,
DET_MASTER_PORT. When starting the agent as a container, environment
variables can be specified as part of
docker run \ -e DET_MASTER_HOST=127.0.0.1 \ -e DET_MASTER_PORT=8080 \ determinedai/determined-agent
The equivalent behavior can be achieved using command-line options:
determined-agent run --master-host=127.0.0.1 --master-port=8080
The same behavior applies to master configuration settings as well. For example, configuring the host where the Postgres database is running can be done via a configuration file containing:
db: host: the-db-host
Equivalent behavior can be achieved by setting the
DET_DB_HOST=the-db-host environment variable or
the-db-host command-line option.
In the rest of this document, we will refer to options using their names
in the configuration file. Periods (
.) will be used to indicate
nested options; for example, the option above would be indicated by
By default, the master listens on TCP port 8080. This can be configured
The master can secure all incoming connections using TLS. That
ability requires a TLS private key and certificate to be provided; set
security.tls.key to paths to a
PEM-encoded TLS certificate and private key, respectively, to do so. If
TLS is enabled, the default port becomes 8443 rather than 8080. See
Transport Layer Security for more information.
Configuring Trial Runner Networking¶
The master is capable of selecting the network interface that trial
runners will use to communicate when performing distributed
(multi-machine) training. The network interface can be configured by
task_container_defaults.dtrain_network_interface. If left
unspecified, which is the default setting, Determined will auto-discover
a common network interface shared by the trial runners.
For Distributed Training, Determined automatically detects a common network interface shared by the agent machines. If your cluster has multiple common network interfaces, please specify the fastest one.
Additionally, the ports used by the GLOO and NCCL libraries, which are used during distributed (multi-machine) training can be configured to fall within user-defined ranges. If left unspecified, ports will be chosen randomly from the unprivileged port range (1024-65535).
Default Checkpoint Storage¶
See Checkpoint Storage Configuration for details.
By default, the master and WebUI collect anonymous information about how Determined is being used. This usage information is collected so that we can improve the design of the product. Determined does not report information that can be used to identify individual users of the product, nor does it include model source code, model architecture/checkpoints, training datasets, training and validation metrics, logs, or hyperparameter values.
The information we collect from the master periodically includes:
a unique, randomly generated ID for the current database and for the current instance of the master
the version of Determined
the version of Go that was used to compile the master
the number of registered users
the number of experiments that have been created
the total number of trials across all experiments
the number of active, paused, completed, and canceled experiments
We also record when the following events happen:
an experiment is created
an experiment’s state changes
an agent connects or disconnects
a user is created (the username is not transmitted)
When an experiment is created, we report:
resourcessections of the experiment config
the name of the container image used
the total number of hyperparameters
the value of the
When an experiment terminates, we report:
the number of trials in the experiment
the total number of training workloads across all trials in the experiment
the total elapsed time for all workloads across all trials in the experiment
The information we collect from the WebUI includes:
pages that are visited
errors that occur (both network errors and uncaught exceptions)
To disable telemetry reporting in both the master and the WebUI, start
the master with the
--telemetry-enabled=false flag (this can also be
done by editing the master config file or setting an environment
variable, as with any other configuration option). Disabling telemetry
reporting will not affect the functionality of Determined in any way.
The Determined master supports a range of configuration settings that
can be set via a YAML configuration file,
environment variables, or command-line options. The configuration file
is normally located at
/etc/determined/master.yaml on the master and
is read when the master starts.
The configuration of an active master can be examined using the
Determined CLI with the command
det master config.
The master supports the following configuration settings:
config_file: Path to the master configuration file. Normally this should only be set via an environment variable or command-line option. Defaults to
scheduler: Specifies how Determined schedules tasks to agents.
fit: The scheduling policy to use when assigning tasks to agents in the cluster. Defaults to
best: The best-fit policy ensures that tasks will be preferentially “packed” together on the smallest number of agents.
worst: The worst-fit policy ensures that tasks will be placed on under-utilized agents.
type: The scheduling policy to use when allocating resources between different tasks (experiments, notebooks, etc.). Defaults to
fair_share: Tasks receive a proportional amount of the available resources depending on the resource they require and their weight.
priority: Tasks are scheduled in the order of the order in which they arrive at the cluster.
resource_provider: The resource provider to use to acquire agents. Defaults to the default resource provider.
type: default: The default resource provider includes static and dynamic agents.
type: kubernetes: The
kubernetesresource provider launches tasks on a Kubernetes cluster. The Determined master must be running within the Kubernetes cluster. When using the
kubernetesresource provider, we recommend deploying Determined using the Determined Helm Chart. When installed via Helm, the configuration settings below will be set automatically.
namespace: The namespace where Determined will deploy Pods and ConfigMaps.
max_slots_per_pod: Each multi-GPU (distributed training) task will be scheduled as a set of
slots_per_task / max_slots_per_podseparate pods, with each pod assigned up to
max_slots_per_podGPUs. Distributed tasks with sizes that are not divisible by
max_slots_per_podare never scheduled. If you have a cluster of different size nodes, set
max_slots_per_podto the smallest common denominator. For example, if you have nodes with 4 GPUs and other nodes with 8 GPUs, set
max_slots_per_podto 4 so that all distributed experiments will launch with 4 GPUs per pod (e.g., on nodes with 8 GPUs, two such pods would be launched).
master_service_name: The service account Determined uses to interact with the Kubernetes API.
port: The TCP port on which the master accepts all incoming connections. Defaults to
task_container_defaults: Specifies Docker defaults for all task containers. A task represents a single schedulable unit, such as a trial, command, or tensorboard.
shm_size_bytes: The size (in bytes) of
/dev/shmfor Determined task containers. Defaults to
network_mode: The Docker network to use for the Determined task containers. If this is set to
host, Docker host-mode networking will be used instead. Defaults to
dtrain_network_interface: The network interface to use during Distributed Training. If not set, Determined automatically determines the network interface to use.
When training a model with multiple machines, the host network interface used by each machine must have the same interface name across machines. The network interface to use can be determined automatically, but there may be issues if there is an interface name common to all machines but it is not routable between machines. Determined already filters out common interfaces like
docker0, but agent machines may have others. If interface detection is not finding the appropriate interface, the
dtrain_network_interfaceoption can be used to set it explicitly (e.g.,
nccl_port_range: The range of ports that NCCL is permitted to use during distributed training. A valid port range is in the format of
MIN:MAX. By default, no restrictions are placed on the NCCL port range.
gloo_port_range: The range of ports that Gloo is permitted to use during distributed training. A valid port range is in the format of
MIN:MAX. By default, no restrictions are placed on the Gloo port range.
cpu_pod_spec: Defines the default pod spec which will be applied to all CPU-only tasks when running on Kubernetes. See Specifying Custom Pod Specs for details.
gpu_pod_spec: Defines the default pod spec which will be applied to all GPU tasks when running on Kubernetes. See Specifying Custom Pod Specs for details.
image: Defines the default docker image to use when executing the workload. If a docker image is specified in the experiment config this default is overriden. This image must be accessible via
docker pullto every Determined agent machine in the cluster. Users can use different container images for GPU vs. CPU agents differently by specifying a dict with two keys,
gpu. Default values:
determinedai/environments:py-3.6.9-pytorch-1.4-tf-1.15-cpu-0.8.0for CPU agents
determinedai/environments:cuda-10.0-pytorch-1.4-tf-1.15-gpu-0.8.0for GPU agents.
force_pull_image: Defines the default policy for forcibly pulling images from the docker registry and bypassing the docker cache. If a pull policy is specified in the experiment config this default value is overriden. Please note that as of November 1st, 2020 unauthenticated users will be capped at 100 pulls from Docker per 6 hours. Defaults to
registry_auth: Defines the default docker registry credentials to use when pulling a custom base docker image, if needed. If credentials are specified is in the experiment config this default value is overriden. Credentials are specified as the following nested fields:
root: Specifies the root directory of the state files. Defaults to
cluster_name(optional): Specify a human readable name for this cluster.
tensorboard_timeout: Specifies the duration in seconds before idle TensorBoard instances are automatically terminated. A TensorBoard instance is considered to be idle if it does not receive any HTTP traffic. The default timeout is
provisioner: Specifies the configuration of dynamic agents.
master_url: The full URL of the master. A valid URL is in the format of
scheme://host:port. The scheme must be either
https. If the master is deployed on EC2, rather than hardcoding the IP address, we advise you use one of the following to set the host as an alias:
public-hostname. If the master is deployed on GCP, rather than hardcoding the IP address, we advise you use one of the following to set the host as an alias:
external-ip. Which one you should select is based on your network configuration. On master startup, we will replace the above alias host with its real value. Defaults to
httpas scheme, local IP address as host, and
master_cert_name: A hostname for which the master’s TLS certificate is valid, if the host specified by the
master_urloption is an IP address or is not contained in the certificate. See Transport Layer Security for more information.
startup_script: One or more shell commands that will be run during agent instance start up. These commands are executed as root as soon as the agent cloud instance has started and before the Determined agent container on the instance is launched. For example, this feature can be used to mount a distributed file system or make changes to the agent instance’s configuration. The default value is the empty string. It may be helpful to use the YAML
|syntax to specify a multi-line string. For example,
startup_script: | mkdir -p /mnt/disks/second mount /dev/sdb1 /mnt/disks/second
container_startup_script: One or more shell commands that will be run when the Determined agent container is started. These commands are executed inside the agent container but before the Determined agent itself is launched. For example, this feature can be used to configure Docker so that the agent can pull task images from GCR securely (see this example for more details). The default value is the empty string.
agent_docker_image: The Docker image to use for the Determined agents. A valid form is
agent_docker_network: The Docker network to use for the Determined agent and task containers. If this is set to
host, Docker host-mode networking will be used instead. The default value is
agent_docker_runtime: The Docker runtime to use for the Determined agent and task containers. Defaults to
max_idle_agent_period: How long to wait before terminating idle dynamic agents. This string is a sequence of decimal numbers, each with optional fraction and a unit suffix, such as “30s”, “1h”, or “1m30s”. Valid time units are “s”, “m”, “h”. The default value is
max_agent_starting_period: How long to wait for agents starting before retrying. This string is a sequence of decimal numbers, each with optional fraction and a unit suffix, such as “30s”, “1h”, or “1m30s”. Valid time units are “s”, “m”, “h”. The default value is
provider: aws: Specifies running dynamic agents on AWS. (Required)
region: The region of the AWS resources used by Determined. We advise setting this region to be the same region as the Determined master for better network performance. Defaults to the same region as the master.
root_volume_size: Size of the root volume of the Determined agent in GB. We recommend at least 100GB. Defaults to
image_id: The AMI ID of the Determined agent. Defaults to the latest GCP agent image. (Optional)
tag_key: Key for tagging the Determined agent instances. Defaults to
tag_value: Value for tagging the Determined agent instances. Defaults to the master instance ID if the master is on EC2, otherwise
instance_name: Name to set for the Determined agent instances. Defaults to
ssh_key_name: The name of the SSH key registered with AWS for SSH key access to the agent instances. (Required)
iam_instance_profile_arn: The Amazon Resource Name (ARN) of the IAM instance profile to attach to the agent instances.
network_interface: Network interface to set for the Determined agent instances.
public_ip: Whether to use public IP addresses for the Determined agents. See Network Requirements for instructions on whether a public IP should be used. Defaults to
security_group_id: The ID of the security group to run the Determined agents as. This should be the security group you identified or created in Network Requirements. Defaults to the default security group of the specified VPC.
subnet_id: The ID of the subnet to run the Determined agents in. Defaults to the default subnet of the default VPC.
max_instances: Maximum number of dynamic agent instances. Defaults to
instance_type: AWS instance type to use for dynamic agents. This must be one of the following:
p3dn.24xlarge. Defaults to
spot: Whether to use spot instances. Defaults to
false. See AWS Spot Instances for more details.
spot_max_price: Optional field indicating the maximum price per hour that you are willing to pay for a spot instance. The market price for a spot instance varies based on supply and demand. If the market price exceeds the
spot_max_price, Determined will not launch instances. This field must be a string and must not include a currency sign. For example, $2.50 should be represented as
"2.50". Defaults to the on-demand price for the given instance type.
provider: gcp: Specifies running dynamic agents on GCP. (Required)
base_config: Instance resource base configuration that will be merged with the fields below to construct GCP inserting instance request. See REST Resource: instances for details.
project: The project ID of the GCP resources used by Determined. Defaults to the project of the master.
zone: The zone of the GCP resources used by Determined. Defaults to the zone of the master.
boot_disk_size: Size of the root volume of the Determined agent in GB. We recommend at least 100GB. Defaults to
boot_disk_source_image: The boot disk source image of the Determined agent that was shared with you. To use a specific version of the Determined agent image from a specific project, it should be set in the format:
projects/<project-id>/global/images/<image-id>. Defaults to the latest GCP agent image. (Optional)
label_key: Key for labeling the Determined agent instances. Defaults to
label_value: Value for labeling the Determined agent instances. Defaults to the master instance name if the master is on GCP, otherwise
name_prefix: Name prefix to set for the Determined agent instances. The names of the Determined agent instances are a concatenation of the name prefix and a pet name. Defaults to the master instance name if the master is on GCP otherwise
network_interface: Network configuration for the Determined agent instances. See the GCP API Access section for the suggested configuration. (Required)
network: Network resource for the Determined agent instances. The network configuration should specify the project ID of the network. It should be set in the format:
subnetwork: Subnetwork resource for the Determined agent instances. The subnet configuration should specify the project ID and the region of the subnetwork. It should be set in the format:
external_ip: Whether to use external IP addresses for the Determined agent instances. See Network Requirements for instructions on whether an external IP should be set. Defaults to
network_tags: An array of network tags to set firewalls for the Determined agent instances. This is the one you identified or created in Firewall Rules. Defaults to be an empty array.
service_account: Service account for the Determined agent instances. See the GCP API Access section for suggested configuration.
scopes: List of scopes authorized for the Determined agent instances. As suggested in GCP API Access, we recommend you set the scopes to
["https://www.googleapis.com/auth/cloud-platform"]. Defaults to
instance_type: Type of instance for the Determined agents.
machine_type: Type of machine for the Determined agents. Defaults to
gpu_type: Type of GPU for the Determined agents. Set it to be an empty string to not use any GPUs. Defaults to
gpu_num: Number of GPUs for the Determined agents. Defaults to 4.
preemptible: Whether to use preemptible instances. Defaults to
min_instances: Min number of Determined agent instances. Defaults to 0.
max_instances: Max number of Determined agent instances. Defaults to 5.
operation_timeout_period: The timeout period for tracking a GCP operation. This string is a sequence of decimal numbers, each with optional fraction and a unit suffix, such as “30s”, “1h”, or “1m30s”. Valid time units are “s”, “m”, “h”. The default value is
checkpoint_storage: Specifies where model checkpoints will be stored. This can be overridden on a per-experiment basis in the Experiment Configuration. A checkpoint contains the architecture and weights of the model being trained. Determined currently supports four kinds of checkpoint storage,
shared_fs, identified by the
type: gcs: Checkpoints are stored on Google Cloud Storage (GCS). Authentication is done using GCP’s “Application Default Credentials” approach. When using Determined inside Google Compute Engine (GCE), the simplest approach is to ensure that the VMs used by Determined are running in a service account that has the “Storage Object Admin” role on the GCS bucket being used for checkpoints. As an alternative (or when running outside of GCE), you can add the appropriate service account credentials to your container (e.g., via a bind-mount), and then set the
GOOGLE_APPLICATION_CREDENTIALSenvironment variable to the container path where the credentials are located. See Environment Variables for more information on how to set environment variables in trial environments.
bucket: The GCS bucket name to use.
type: hdfs: Checkpoints are stored in HDFS using the WebHDFS API for reading and writing checkpoint resources.
hdfs_url: Hostname or IP address of HDFS namenode, prefixed with protocol, followed by WebHDFS port on namenode. Multiple namenodes are allowed as a semicolon-separated list (e.g.,
hdfs_path: The prefix path where all checkpoints will be written to and read from. The resources of each checkpoint will be saved in a subdirectory of
hdfs_path, where the subdirectory name is the checkpoint’s UUID.
user: An optional string value that indicates the user to use for all read and write requests. If left unspecified, the default user of the trial runner container will be used.
type: s3: Checkpoints are stored in Amazon S3.
bucket: The S3 bucket name to use.
access_key: The AWS access key to use.
secret_key: The AWS secret key to use.
endpoint_url: The optional endpoint to use for S3 clones, e.g.,
type: shared_fs: Checkpoints are written to a directory on the agent’s file system. The assumption is that the system administrator has arranged for the same directory to be mounted at every agent host, and for the content of this directory to be the same on all agent hosts (e.g., by using a distributed or network file system such as GlusterFS or NFS).
host_path: The file system path on each agent to use. This directory will be mounted to
/determined_shared_fsinside the trial container.
storage_path: The optional path where checkpoints will be written to and read from. Must be a subdirectory of the
host_pathor an absolute path containing the
host_path. If unset, checkpoints are written to and read from the
propagation: (Advanced users only) Optional propagation behavior for replicas of the bind-mount. Defaults to
When an experiment finishes, the system will optionally delete some checkpoints to reclaim space. The
save_trial_latestparameters specify which checkpoints to save. See Checkpoint Garbage Collection for more details.
db: Specifies the configuration of the database.
user: The database user to use when logging in the database. (Required)
password: The password to use when logging in the database. (Required)
host: The database host to use. (Required)
port: The database port to use. (Required)
name: The database name to use. (Required)
security: Specifies security-related configuration settings.
tls: Specifies configuration settings for TLS. TLS is enabled if certificate and key files are both specified.
cert: Certificate file to use for serving TLS.
key: Key file to use for serving TLS.
telemetry: Specifies whether we collect and report anonymous information about the usage of Determined. See Telemetry for details on what kinds of information are reported.
enabled: Whether telemetry is enabled. Defaults to
config_file: Path to the agent configuration file. Normally this should only be set via an environment variable or command-line option. Defaults to
master_host(required): The hostname or IP address of the Determined master.
master_port: The port of the Determined master. Defaults to
443if TLS is enabled and
agent_id: The ID of this agent; defaults to the hostname of the current machine. Agent IDs must be unique within a cluster.
container-master-host: Master hostname that containers started by this agent will connect to. Defaults to the value of
container-master-port: Master port that containers started by this agent will connect to. Defaults to the value of
label: The label to assign to this agent. An agent with a label will only be assigned workloads that have been assigned the same label (e.g., via the agent_label field in the experiment configuration).
visible_gpus: The GPUs that should be exposed as slots by the agent. A comma-separated list of GPUs, each specified by a 0-based index, UUID, PCI bus ID, or board serial number. The 0-based index of NVIDIA GPUs can be obtained via the
slot_type: The slot type that should be exposed. Dynamic agents having GPUs will be configured to
gpuwhile those agents having no GPUs will be configured to
none. For static agents this field defaults to
auto: Automatically detects the slot type. The agent will detect if there are GPUs. If there are GPUs, it maps each GPU to one slot. Otherwise, it maps all the CPUs to a slot.
none: The agent will not create any slots for detected devices.
gpu: The agent will map each detected GPU to a slot.
http_proxy: The HTTP proxy address for the agent’s containers.
https_proxy: The HTTPS proxy address for the agent’s containers.
ftp_proxy: The FTP proxy address for the agent’s containers.
no_proxy: The addresses that the agent’s containers should not proxy.
security: Security-related configuration settings.
tls: Configuration settings for TLS.
enabled: Whether to use TLS to connect to the master. Defaults to
skip_verify: Skip verifying the master certificate when using TLS. Defaults to
false. Enabling this setting will reduce the security of your Determined cluster.
master_cert: CA cert file for the master when using TLS.
master_cert_name: A hostname for which the master’s TLS certificate is valid, if the value of the
master_hostoption is an IP address or is not contained in the certificate.