Shortcuts

Command-line Interface

The Determined command-line interface (CLI) is installed under the name det. Most uses of it require arguments specifying a type of object to act on and the action to perform, followed by any additional arguments or options. For example,

det experiment config 17

displays the configuration for experiment 17. Some commands have an additional level of nesting:

det experiment label add 17 foobar

adds the label “foobar” to experiment 17.

The names of some objects, actions, and options may be abbreviated; alternative names are shown in the full reference in parentheses. For example, these two commands are equivalent:

det -u admin experiment list
det --user admin experiment list

as are these two:

det e config 17
det experiment config 17

Additionally, when list is a valid action, it is the default when no action is specified, so the following pairs of commands are also equivalent:

det e
det experiment list
det s
det slot list

Commands that launch Determined tasks can take configuration options that change the behavior of the tasks. These configurations affect Determined notebooks, shells, commands, and TensorBoards.

Providing -h or --help as an argument anywhere will cause the CLI to exit after printing help text for the object or action specified up to that point.

Full Listing

Determined command-line client

usage: det [-h] [-u username] [-m address] [-v] command ...

Positional Arguments

command

Possible choices: help, auth, agent, a, command, cmd, checkpoint, c, deploy, d, experiment, e, master, m, model, notebook, oauth, preview-search, resources, res, shell, slot, s, task, template, tpl, tensorboard, trial, t, user, u, version

Named Arguments

-u, --user

run as the given user

-m, --master

master address

Default: “localhost:8080”

-v, --version

print CLI version and exit

Sub-commands:

help

show help for this command

det help [-h]

auth

manage auth

det auth [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, list-providers, login

Sub-commands:
help

show help for this command

det auth help [-h]
list-providers

lists the available auth providers

det auth list-providers [-h]
login

sign on with an auth provider

det auth login [-h] [-p PROVIDER]
Named Arguments
-p, --provider

auth provider to use (not needed if the Determined master only supports one provider)

agent (a)

manage agents

det agent [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, disable, enable, list

Sub-commands:
help

show help for this command

det agent help [-h]
disable

disable agent

det agent disable [-h] [--all] [agent_id]
Positional Arguments
agent_id

agent ID

Named Arguments
--all

disable all agents

Default: False

enable

enable agent

det agent enable [-h] [--all] [agent_id]
Positional Arguments
agent_id

agent ID

Named Arguments
--all

enable all agents

Default: False

list

list agents

det agent list [-h] [--csv | --json]
Named Arguments
--csv

print as CSV

Default: False

--json

print as JSON

Default: False

command (cmd)

manage commands

det command [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, config, kill, list, ls, logs, run

Sub-commands:
help

show help for this command

det command help [-h]
config

display command config

det command config [-h] id
Positional Arguments
id

command ID

kill

forcibly terminate a command

det command kill [-h] [-f] command_id [command_id ...]
Positional Arguments
command_id

command ID

Named Arguments
-f, --force

ignore errors

Default: False

list (ls)

list commands

det command list [-h] [-q] [--all]
Named Arguments
-q, --quiet

only display the IDs

Default: False

--all, -a

show all commands (including other users’)

Default: False

logs

fetch command logs

det command logs [-h] [-f] [--tail TAIL] command_id
Positional Arguments
command_id

command ID

Named Arguments
-f, --follow

follow the logs of a command, similar to tail -f

Default: False

--tail

number of lines to show, counting from the end of the log

Default: 200

run

create command

det command run [-h] [--config-file CONFIG_FILE] [-v VOLUME] [-c CONTEXT]
                [--config CONFIG] [--template TEMPLATE] [-d]
                ...
Positional Arguments
entrypoint

entrypoint command and arguments to execute

Named Arguments
--config-file

command config file (.yaml)

-v, --volume

A mount specification in the form of <host path>:<container path>. The given path on the host machine will be mounted under the given path in the command container.

Default: []

-c, --context

The filepath to a directory that contains the set of files used to execute the command. All files under this directory will be packaged, maintaining the existing directory structure. The total byte contents of the directory must not exceed 96 MB. By default, the context directory will be empty.

--config

Additional configuration arguments for setting up a command. Arguments should be specified as key=value. Nested configuration keys can be specified by dot notation, e.g., resources.slots=4. List values can be specified by comma-separated values.

Default: []

--template

name of template to apply to the command configuration

-d, --detach

run in the background and print the ID

Default: False

checkpoint (c)

manage checkpoints

det checkpoint [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, describe, download

Sub-commands:
help

show help for this command

det checkpoint help [-h]
describe

describe checkpoint

det checkpoint describe [-h] uuid
Positional Arguments
uuid

checkpoint uuid to describe

download

download checkpoint from persistent storage

det checkpoint download [-h] [-o OUTPUT_DIR] [-q] uuid
Positional Arguments
uuid

Download a checkpoint by specifying its UUID.

Named Arguments
-o, --output-dir

Desired output directory for the checkpoint.

-q, --quiet

Only print the path to the checkpoint.

Default: False

deploy (d)

manage deployments

det deploy [-h] [--version] [--no-preflight-checks] [--no-wait-for-master]
           [--image-repo-prefix IMAGE_REPO_PREFIX]
           subcommand ...
Positional Arguments
subcommand

Possible choices: help, aws, gcp, local

Named Arguments
--version

show program’s version number and exit

--no-preflight-checks

Disable preflight checks

Default: False

--no-wait-for-master

Do not wait for master to come up after AWS or GCP clusters are deployed

Default: False

--image-repo-prefix

Docker image repository to use for determined-master and determined-agent images

Default: “determinedai”

Sub-commands:
help

show help for this command

det deploy help [-h]
aws

AWS help

det deploy aws [-h] subsubcommand ...
Positional Arguments
subsubcommand

Possible choices: help, down, list, up

Sub-commands:
help

show help for this command

det deploy aws help [-h]
down

delete CloudFormation stack

det deploy aws down [-h] --cluster-id CLUSTER_ID [--region REGION]
                    [--profile PROFILE]
Named Arguments
--region

AWS region

--profile

AWS profile

required named arguments
--cluster-id

stack name for CloudFormation cluster

list

list CloudFormation stacks

det deploy aws list [-h] [--region REGION] [--profile PROFILE]
Named Arguments
--region

AWS region

--profile

AWS profile

up

deploy/update CloudFormation stack

det deploy aws up [-h] --cluster-id CLUSTER_ID --keypair KEYPAIR
                  [--region REGION] [--profile PROFILE]
                  [--master-instance-type MASTER_INSTANCE_TYPE]
                  [--enable-cors] [--master-tls-cert MASTER_TLS_CERT]
                  [--master-tls-key MASTER_TLS_KEY]
                  [--master-cert-name MASTER_CERT_NAME]
                  [--compute-agent-instance-type COMPUTE_AGENT_INSTANCE_TYPE]
                  [--aux-agent-instance-type AUX_AGENT_INSTANCE_TYPE]
                  [--deployment-type {simple,secure,vpc,efs,fsx,govcloud}]
                  [--inbound-cidr INBOUND_CIDR]
                  [--agent-subnet-id AGENT_SUBNET_ID]
                  [--db-password DB_PASSWORD]
                  [--max-idle-agent-period MAX_IDLE_AGENT_PERIOD]
                  [--max-agent-starting-period MAX_AGENT_STARTING_PERIOD]
                  [--max-aux-containers-per-agent MAX_AUX_CONTAINERS_PER_AGENT]
                  [--min-dynamic-agents MIN_DYNAMIC_AGENTS]
                  [--max-dynamic-agents MAX_DYNAMIC_AGENTS] [--spot]
                  [--spot-max-price SPOT_MAX_PRICE]
                  [--scheduler-type SCHEDULER_TYPE]
                  [--preemption-enabled PREEMPTION_ENABLED] [--dry-run]
                  [--cpu-env-image CPU_ENV_IMAGE]
                  [--gpu-env-image GPU_ENV_IMAGE]
                  [--log-group-prefix LOG_GROUP_PREFIX] [--retain-log-group]
Named Arguments
--region

AWS region

--profile

AWS profile

--master-instance-type

instance type for master

--enable-cors

allow CORS requests or not: true/false

Default: False

--master-tls-cert
--master-tls-key
--master-cert-name
--compute-agent-instance-type, --gpu-agent-instance-type

instance type for agent in the compute (previously, GPU) resource pool

--aux-agent-instance-type, --cpu-agent-instance-type

instance type for agent in the auxiliary (previously, CPU) resource pool

--deployment-type

Possible choices: simple, secure, vpc, efs, fsx, govcloud

deployment type - must be one of [simple, secure, vpc, efs, fsx, govcloud]

Default: “simple”

--inbound-cidr

inbound IP Range in CIDR format

--agent-subnet-id

subnet to deploy agents into. Optional. Only used with simple deployment type

--db-password

password for master database

Default: “postgres”

--max-idle-agent-period

max agent idle time

--max-agent-starting-period

max agent starting time

--max-aux-containers-per-agent, --max-cpu-containers-per-agent

maximum number of containers on agent in the auxiliary (previously, CPU) resource pool

--min-dynamic-agents

minimum number of dynamic agent instances at one time

--max-dynamic-agents

maximum number of dynamic agent instances at one time

--spot

whether to use spot instances or not

Default: False

--spot-max-price

maximum hourly price for the spot instance (do not include the dollar sign)

--scheduler-type

scheduler to use (defaults to fair_share).

Default: fair_share

--preemption-enabled

whether task preemption is supported in the scheduler (only configurable for priority scheduler).

Default: “false”

--dry-run

print deployment template

Default: False

--cpu-env-image

Docker image for CPU tasks

--gpu-env-image

Docker image for GPU tasks

--log-group-prefix

prefix for output CloudWatch log group

--retain-log-group

whether to retain CloudWatch log group after the stack is deleted (only available for the simple template)

required named arguments
--cluster-id

stack name for CloudFormation cluster

--keypair

aws ec2 keypair for master and agent

gcp

gcp_help

det deploy gcp [-h] subsubcommand ...
Positional Arguments
subsubcommand

Possible choices: help, down, up

Sub-commands:
help

show help for this command

det deploy gcp help [-h]
down

delete gcp cluster

det deploy gcp down [-h] [--local-state-path LOCAL_STATE_PATH] [--no-prompt]
optional named arguments
--local-state-path

local directory for storing cluster state

Default: “/home/circleci/project/docs”

--no-prompt

no prompt when deleting resources

Default: False

up

create gcp cluster

det deploy gcp up [-h] --cluster-id CLUSTER_ID --project-id PROJECT_ID
                  [--dry-run] [--keypath KEYPATH] [--network NETWORK]
                  [--filestore-address FILESTORE_ADDRESS] [--no-filestore]
                  [--region REGION] [--zone ZONE]
                  [--local-state-path LOCAL_STATE_PATH]
                  [--preemptible PREEMPTIBLE]
                  [--operation-timeout-period OPERATION_TIMEOUT_PERIOD]
                  [--master-instance-type MASTER_INSTANCE_TYPE]
                  [--compute-agent-instance-type COMPUTE_AGENT_INSTANCE_TYPE]
                  [--aux-agent-instance-type AUX_AGENT_INSTANCE_TYPE]
                  [--db-password DB_PASSWORD]
                  [--max-aux-containers-per-agent MAX_AUX_CONTAINERS_PER_AGENT]
                  [--max-idle-agent-period MAX_IDLE_AGENT_PERIOD]
                  [--max-agent-starting-period MAX_AGENT_STARTING_PERIOD]
                  [--port PORT] [--gpu-type GPU_TYPE] [--gpu-num GPU_NUM]
                  [--min-dynamic-agents MIN_DYNAMIC_AGENTS]
                  [--max-dynamic-agents MAX_DYNAMIC_AGENTS]
                  [--min-cpu-platform-master MIN_CPU_PLATFORM_MASTER]
                  [--min-cpu-platform-agent MIN_CPU_PLATFORM_AGENT]
                  [--scheduler-type SCHEDULER_TYPE]
                  [--preemption-enabled PREEMPTION_ENABLED]
                  [--cpu-env-image CPU_ENV_IMAGE]
                  [--gpu-env-image GPU_ENV_IMAGE]
required named arguments
--cluster-id

unique identifier to name and tag resources

--project-id

project ID to create the cluster in

optional named arguments
--dry-run

return the infrastructure plan to be executed based on your arguments

Default: False

--keypath

path to service account key if not using default credentials

--network

network name to create (the network should not already exist in the project)

Default: “det-default”

--filestore-address

the address of an existing Filestore in the format of ‘ip-address:/file-share’; if not provided, a new Filestore instance will be created

Default: “”

--no-filestore

whether to create a new Filestore if no filestore address is provided

Default: False

--region

region to create the cluster in (defaults to us-west1)

Default: “us-west1”

--zone

zone to create the cluster in (defaults to region-b)

--local-state-path

local directory for storing cluster state

Default: “/home/circleci/project/docs”

--preemptible

whether to use preemptible instances for dynamic agents

Default: False

--operation-timeout-period

operation timeout before retrying, e.g. 5m for 5 minutes

Default: “5m”

--master-instance-type

instance type for master

Default: “n1-standard-2”

--compute-agent-instance-type, --gpu-agent-instance-type

instance type for agent in the compute (previously, GPU) resource pool

Default: “n1-standard-32”

--aux-agent-instance-type, --cpu-agent-instance-type

instance type for agent in the auxiliary (previously, CPU) resource pool

Default: “n1-standard-4”

--db-password

password for master database

Default: “postgres”

--max-aux-containers-per-agent, --max-cpu-containers-per-agent

maximum number of containers on agent in the auxiliary (previously, CPU) resource pool

Default: 100

--max-idle-agent-period

max agent idle time before it is shut down, e.g. 30m for 30 minutes

Default: “10m”

--max-agent-starting-period

max agent starting time before retrying, e.g. 30m for 30 minutes

Default: “20m”

--port

port to use for communication on master instance

Default: 8080

--gpu-type

type of GPU to use on agents

Default: “nvidia-tesla-k80”

--gpu-num

number of GPUs per agent instance

Default: 8

--min-dynamic-agents

minimum number of dynamic agent instances at one time

Default: 0

--max-dynamic-agents

maximum number of dynamic agent instances at one time

Default: 5

--min-cpu-platform-master

minimum cpu platform for master instances

Default: “Intel Skylake”

--min-cpu-platform-agent

minimum cpu platform for agent instances

Default: “Intel Broadwell”

--scheduler-type

scheduler to use (defaults to fair_share).

Default: fair_share

--preemption-enabled

whether task preemption is supported in the scheduler (only configurable for priority scheduler).

Default: False

--cpu-env-image

Docker image for CPU tasks

Default: “”

--gpu-env-image

Docker image for GPU tasks

Default: “”

local

local help

det deploy local [-h] subsubcommand ...
Positional Arguments
subsubcommand

Possible choices: help, agent-down, agent-up, cluster-down, cluster-up, logs, master-down, master-up

Sub-commands:
help

show help for this command

det deploy local help [-h]
agent-down

Stop a Determined agent

det deploy local agent-down [-h] [--agent-name AGENT_NAME] [--all]
                            [--cluster-name CLUSTER_NAME]
Named Arguments
--agent-name

agent name

Default: “det-agent”

--all

stop all running agents

Default: False

--cluster-name

name for the cluster resources

Default: “determined”

agent-up

Start a Determined agent

det deploy local agent-up [-h] [--master-port MASTER_PORT]
                          [--det-version DET_VERSION]
                          [--agent-name AGENT_NAME]
                          [--agent-label AGENT_LABEL]
                          [--agent-resource-pool AGENT_RESOURCE_POOL] [--gpu]
                          [--no-gpu] [--no-autorestart]
                          [--cluster-name CLUSTER_NAME]
                          master_host
Positional Arguments
master_host

master hostname

Named Arguments
--master-port

master port

Default: 8080

--det-version

version or commit to use

--agent-name

agent name

Default: “det-agent”

--agent-label

agent label

--agent-resource-pool

agent resource pool

--gpu

enable GPU support for agent

Default: True

--no-gpu

disable GPU support for agent

Default: True

--no-autorestart

disable container auto-restart (recommended for local development)

Default: False

--cluster-name

name for the cluster resources

Default: “determined”

cluster-down

Stop a Determined cluster

det deploy local cluster-down [-h] [--cluster-name CLUSTER_NAME] [--delete-db]
Named Arguments
--cluster-name

name for the cluster resources

Default: “determined”

--delete-db

remove current master database

Default: False

cluster-up

Create a Determined cluster

det deploy local cluster-up [-h]
                            [--master-config-path MASTER_CONFIG_PATH | --storage-host-path STORAGE_HOST_PATH]
                            [--agents AGENTS] [--master-port MASTER_PORT]
                            [--cluster-name CLUSTER_NAME]
                            [--det-version DET_VERSION]
                            [--db-password DB_PASSWORD] [--delete-db] [--gpu]
                            [--no-gpu] [--no-autorestart]
                            [--auto-bind-mount AUTO_BIND_MOUNT]
                            [--no-auto-bind-mount]
Named Arguments
--master-config-path

path to master configuration

--storage-host-path

Storage location for cluster data (e.g. checkpoints)

Default: /home/circleci/.local/share/determined

--agents

number of agents to start (on this machine)

Default: 1

--master-port

port to expose master on

Default: 8080

--cluster-name

name for the cluster resources

Default: “determined”

--det-version

version or commit to use

--db-password

password for master database

Default: “postgres”

--delete-db

remove current master database

Default: False

--gpu

enable GPU support for agent

Default: True

--no-gpu

disable GPU support for agent

Default: True

--no-autorestart

disable container auto-restart (recommended for local development)

Default: False

--auto-bind-mount

directory to mount into task containers (default: user’s home directory)

--no-auto-bind-mount

disable mounting user’s home directory into task containers

Default: False

logs

Show the logs of a Determined cluster

det deploy local logs [-h] [--cluster-name CLUSTER_NAME] [--no-follow]
Named Arguments
--cluster-name

name for the cluster resources

Default: “determined”

--no-follow

disable following logs

Default: False

master-down

Stop a Determined master

det deploy local master-down [-h] [--master-name MASTER_NAME] [--delete-db]
                             [--cluster-name CLUSTER_NAME]
Named Arguments
--master-name

name for the cluster resources

Default: “determined”

--delete-db

remove current master database

Default: False

--cluster-name

name for the cluster resources

Default: “determined”

master-up

Start a Determined master

det deploy local master-up [-h]
                           [--master-config-path MASTER_CONFIG_PATH | --storage-host-path STORAGE_HOST_PATH]
                           [--master-port MASTER_PORT]
                           [--master-name MASTER_NAME]
                           [--det-version DET_VERSION]
                           [--db-password DB_PASSWORD] [--delete-db]
                           [--no-autorestart]
                           [--auto-bind-mount AUTO_BIND_MOUNT]
                           [--no-auto-bind-mount]
                           [--cluster-name CLUSTER_NAME]
Named Arguments
--master-config-path

path to master configuration

--storage-host-path

Storage location for cluster data (e.g. checkpoints)

Default: /home/circleci/.local/share/determined

--master-port

port to expose master on

Default: 8080

--master-name

name for the cluster resources

Default: “determined”

--det-version

version or commit to use

--db-password

password for master database

Default: “postgres”

--delete-db

remove current master database

Default: False

--no-autorestart

disable container auto-restart (recommended for local development)

Default: False

--auto-bind-mount

directory to mount into task containers (default: user’s home directory)

Default: “/home/circleci”

--no-auto-bind-mount

disable mounting user’s home directory into task containers

Default: False

--cluster-name

name for the cluster resources

Default: “determined”

experiment (e)

manage experiments

det experiment [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, activate, archive, cancel, config, create, delete, describe, download, download-model-def, kill, label, list, list-checkpoints, lc, list-trials, lt, pause, set, unarchive, wait

Sub-commands:
help

show help for this command

det experiment help [-h]
activate

activate experiment

det experiment activate [-h] experiment_id
Positional Arguments
experiment_id

experiment ID to activate

archive

archive experiment

det experiment archive [-h] experiment_id
Positional Arguments
experiment_id

experiment ID to archive

cancel

cancel experiment

det experiment cancel [-h] experiment_id
Positional Arguments
experiment_id

experiment ID to cancel

config

display experiment config

det experiment config [-h] experiment_id
Positional Arguments
experiment_id

experiment ID

create

create experiment

det experiment create [-h] [-g] [--local] [--template TEMPLATE]
                      [-f | --paused | -t]
                      config_file model_def
Positional Arguments
config_file

experiment config file (.yaml)

model_def

file or directory containing model definition

Named Arguments
-g, --git

Associate git metadata with this experiment. This flag assumes that git is installed, a .git repository exists in the model definition directory, and that the git working tree of that repository is empty.

Default: False

--local

Create the experiment in local mode instead of submitting it to the cluster. For more information, see documentation on det.experimental.create()

Default: False

--template

name of template to apply to the experiment configuration

-f, --follow-first-trial

follow the logs of the first trial that is created

Default: False

--paused

do not activate the experiment

Default: False

-t, --test-mode

Test the experiment configuration and model definition by creating and scheduling a very small experiment. This command will verify that a training workload and validation workload run successfully and that checkpoints can be saved. The test experiment will be archived on creation.

Default: False

delete

delete experiment

det experiment delete [-h] [--yes] experiment_id
Positional Arguments
experiment_id

delete experiment

Named Arguments
--yes

automatically answer yes to prompts

Default: False

describe

describe experiment

det experiment describe [-h] [--metrics] [--csv | --json | --outdir OUTDIR]
                        experiment_ids
Positional Arguments
experiment_ids

comma-separated list of experiment IDs to describe

Named Arguments
--metrics

display full metrics

Default: False

--csv

print as CSV

Default: False

--json

print as JSON

Default: False

--outdir

directory to save output

download

download checkpoints for an experiment

det experiment download [-h] [-o OUTPUT_DIR] [--top-n TOP_N]
                        [--sort-by SORT_BY]
                        [--smaller-is-better SMALLER_IS_BETTER] [-q]
                        experiment_id
Positional Arguments
experiment_id

experiment ID to download

Named Arguments
-o, --output-dir

Desired top level directory for the checkpoints. Checkpoints will be downloaded to <output_dir>/<checkpoint_uuid>/<checkpoint_files>.

Default: “checkpoints”

--top-n

The number of checkpoints to download for the experiment. The checkpoints are sorted by validation metric as defined by –sort-by and –smaller-is-better.This command will select the best N checkpoints from the top performing N trials of the experiment.

Default: 1

--sort-by

The name of the validation metric to sort on. Without –sort-by, the experiment’s searcher metric is assumed. If this argument is specified, –smaller-is-better must also be specified.

--smaller-is-better

The sort order for metrics when using –sort-by. For example, ‘accuracy’ would require passing ‘–smaller-is-better false’. If –sort-by is specified, this argument must be specified.

-q, --quiet

Only print the paths to the checkpoints.

Default: False

download-model-def

download model definition

det experiment download-model-def [-h] [--output-dir OUTPUT_DIR] experiment_id
Positional Arguments
experiment_id

experiment ID

Named Arguments
--output-dir

output directory

Default: .

kill

kill experiment

det experiment kill [-h] experiment_id
Positional Arguments
experiment_id

experiment ID

label

manage experiment labels

det experiment label [-h] subsubcommand ...
Positional Arguments
subsubcommand

Possible choices: help, add, remove

Sub-commands:
help

show help for this command

det experiment label help [-h]
add

add label

det experiment label add [-h] experiment_id label
Positional Arguments
experiment_id

experiment ID

label

label

remove

remove label

det experiment label remove [-h] experiment_id label
Positional Arguments
experiment_id

experiment ID

label

label

list

list experiments

det experiment list [-h] [--all] [--csv]
Named Arguments
--all, -a

show all experiments (including archived and other users’)

Default: False

--csv

print as CSV

Default: False

list-checkpoints (lc)

list checkpoints of experiment

det experiment list-checkpoints [-h] [--best BEST] [--csv] experiment_id
Positional Arguments
experiment_id

experiment ID

Named Arguments
--best

Return the best N checkpoints for this experiment. If this flag is used, only checkpoints with an associated validation metric will be considered.

--csv

print as CSV

Default: False

list-trials (lt)

list trials of experiment

det experiment list-trials [-h] [--csv] experiment_id
Positional Arguments
experiment_id

experiment ID

Named Arguments
--csv

print as CSV

Default: False

pause

pause experiment

det experiment pause [-h] experiment_id
Positional Arguments
experiment_id

experiment ID to pause

set

set experiment attributes

det experiment set [-h] subsubcommand ...
Positional Arguments
subsubcommand

Possible choices: help, description, gc-policy, max-slots, name, weight

Sub-commands:
help

show help for this command

det experiment set help [-h]
description

set experiment description

det experiment set description [-h] experiment_id description
Positional Arguments
experiment_id

experiment ID to modify

description

experiment description

gc-policy

set experiment GC policy and run GC

det experiment set gc-policy [-h] --save-experiment-best SAVE_EXPERIMENT_BEST
                             --save-trial-best SAVE_TRIAL_BEST
                             --save-trial-latest SAVE_TRIAL_LATEST [--yes]
                             experiment_id
Positional Arguments
experiment_id

experiment ID to modify

Named Arguments
--save-experiment-best

number of best checkpoints per experiment to save

--save-trial-best

number of best checkpoints per trial to save

--save-trial-latest

number of latest checkpoints per trial to save

--yes

automatically answer yes to prompts

Default: False

max-slots

set max_slots of experiment

det experiment set max-slots [-h] experiment_id max_slots
Positional Arguments
experiment_id

experiment ID to modify

max_slots

max slots

name

set experiment name

det experiment set name [-h] experiment_id name
Positional Arguments
experiment_id

experiment ID to modify

name

experiment name

weight

set weight of experiment

det experiment set weight [-h] experiment_id weight
Positional Arguments
experiment_id

experiment ID to modify

weight

weight

unarchive

unarchive experiment

det experiment unarchive [-h] experiment_id
Positional Arguments
experiment_id

experiment ID to unarchive

wait

wait for experiment to reach terminal state

det experiment wait [-h] [--polling-interval POLLING_INTERVAL] experiment_id
Positional Arguments
experiment_id

experiment ID

Named Arguments
--polling-interval

the interval (in seconds) to poll for updated state

Default: 5

master

manage master

det master [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, config, logs

Sub-commands:
help

show help for this command

det master help [-h]
config

fetch master config as JSON

det master config [-h]
logs

fetch master logs

det master logs [-h] [-f] [--tail TAIL]
Named Arguments
-f, --follow

follow the logs of master, similar to tail -f

Default: False

--tail

number of lines to show, counting from the end of the log (default is all)

m (model)

Undocumented

det m [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, create, describe, list, list-versions, register-version

Sub-commands:
help

show help for this command

det m help [-h]
create

create model

det m create [-h] [--description DESCRIPTION] [--json] name
Positional Arguments
name

unique name of the model

Named Arguments
--description

description of the model

--json

print as JSON

Default: False

describe

describe model

det m describe [-h] [--json] [--version VERSION] name
Positional Arguments
name

model to describe

Named Arguments
--json

print as JSON

Default: False

--version

model version information to include in output

Default: 0

list

list all models in the registry

det m list [-h] [--sort-by {name,description,creation_time,last_updated_time}]
           [--order-by {asc,desc}] [--json]
Named Arguments
--sort-by

Possible choices: name, description, creation_time, last_updated_time

sort models by the given field

Default: “last_updated_time”

--order-by

Possible choices: asc, desc

order models in either ascending or descending order

Default: “asc”

--json

print as JSON

Default: False

list-versions

list the versions of a model

det m list-versions [-h] [--json] name
Positional Arguments
name

unique name of the model

Named Arguments
--json

print as JSON

Default: False

register-version

register a new version of a model

det m register-version [-h] [--json] name uuid
Positional Arguments
name

name of the model

uuid

uuid to register as the next version of the model

Named Arguments
--json

print as JSON

Default: False

notebook

manage notebooks

det notebook [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, config, kill, list, ls, logs, open, start

Sub-commands:
help

show help for this command

det notebook help [-h]
config

display notebook config

det notebook config [-h] id
Positional Arguments
id

notebook ID

kill

kill a notebook

det notebook kill [-h] [-f] notebook_id [notebook_id ...]
Positional Arguments
notebook_id

notebook ID

Named Arguments
-f, --force

ignore errors

Default: False

list (ls)

list notebooks

det notebook list [-h] [-q] [--all]
Named Arguments
-q, --quiet

only display the IDs

Default: False

--all, -a

show all notebooks (including other users’)

Default: False

logs

fetch notebook logs

det notebook logs [-h] [-f] [--tail TAIL] notebook_id
Positional Arguments
notebook_id

notebook ID

Named Arguments
-f, --follow

follow the logs of a notebook, similar to tail -f

Default: False

--tail

number of lines to show, counting from the end of the log

Default: 200

open

open an existing notebook

det notebook open [-h] notebook_id
Positional Arguments
notebook_id

notebook ID

start

start a new notebook

det notebook start [-h] [--config-file CONFIG_FILE] [-v VOLUME] [-c CONTEXT]
                   [--config CONFIG] [--template TEMPLATE] [--no-browser] [-d]
                   [--preview]
Named Arguments
--config-file

command config file (.yaml)

-v, --volume

A mount specification in the form of <host path>:<container path>. The given path on the host machine will be mounted under the given path in the command container.

Default: []

-c, --context

The filepath to a directory that contains the set of files used to execute the command. All files under this directory will be packaged, maintaining the existing directory structure. The total byte contents of the directory must not exceed 96 MB. By default, the context directory will be empty.

--config

Additional configuration arguments for setting up a command. Arguments should be specified as key=value. Nested configuration keys can be specified by dot notation, e.g., resources.slots=4. List values can be specified by comma-separated values.

Default: []

--template

name of template to apply to the notebook configuration

--no-browser

don’t open the notebook in a browser after startup

Default: False

-d, --detach

run in the background and print the ID

Default: False

--preview

preview the notebook configuration

Default: False

oauth

manage OAuth

det oauth [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, client

Sub-commands:
help

show help for this command

det oauth help [-h]
client

manage clients

det oauth client [-h] subsubcommand ...
Positional Arguments
subsubcommand

Possible choices: help, add, list, remove

Sub-commands:
help

show help for this command

det oauth client help [-h]
add

add OAuth client application

det oauth client add [-h] name domain
Positional Arguments
name

descriptive name

domain

redirect domain

list

list OAuth client applications

det oauth client list [-h]
remove

remove OAuth client application

det oauth client remove [-h] client_id
Positional Arguments
client_id

OAuth client ID to remove

resources (res)

query historical resource allocation

det resources [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, aggregated, agg, raw

Sub-commands:
help

show help for this command

det resources help [-h]
aggregated (agg)

get aggregated allocation information

det resources aggregated [-h] [--json] [--monthly] start_date end_date
Positional Arguments
start_date

first date to include

end_date

last date to include

Named Arguments
--json

output JSON rather than CSV

Default: False

--monthly

aggregate by month rather than by day

Default: False

raw

get raw allocation information

det resources raw [-h] [--json] timestamp_after timestamp_before
Positional Arguments
timestamp_after
timestamp_before
Named Arguments
--json

output JSON rather than CSV

Default: False

shell

manage shells

det shell [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, config, kill, list, logs, open, show_ssh_command, start

Sub-commands:
help

show help for this command

det shell help [-h]
config

display shell config

det shell config [-h] id
Positional Arguments
id

shell ID

kill

kill a shell

det shell kill [-h] [-f] shell_id [shell_id ...]
Positional Arguments
shell_id

shell ID

Named Arguments
-f, --force

ignore errors

Default: False

list

list shells

det shell list [-h] [-q] [--all]
Named Arguments
-q, --quiet

only display the IDs

Default: False

--all, -a

show all shells (including other users’)

Default: False

logs

fetch shell logs

det shell logs [-h] [-f] [--tail TAIL] shell_id
Positional Arguments
shell_id

shell ID

Named Arguments
-f, --follow

follow the logs of a shell, similar to tail -f

Default: False

--tail

number of lines to show, counting from the end of the log

Default: 200

open

open an existing shell

det shell open [-h] [--show-ssh-command] shell_id [ssh_opts [ssh_opts ...]]
Positional Arguments
shell_id

shell ID

ssh_opts

additional SSH options when connecting to the shell

Named Arguments
--show-ssh-command

show ssh command (e.g. for use in IDE) when starting the shell

Default: False

show_ssh_command

print the ssh command

det shell show_ssh_command [-h] shell_id [ssh_opts [ssh_opts ...]]
Positional Arguments
shell_id

shell ID

ssh_opts

additional SSH options when connecting to the shell

start

start a new shell

det shell start [-h] [--config-file CONFIG_FILE] [-v VOLUME] [-c CONTEXT]
                [--config CONFIG] [-p] [--template TEMPLATE] [-d]
                [--show-ssh-command]
                [ssh_opts [ssh_opts ...]]
Positional Arguments
ssh_opts

additional SSH options when connecting to the shell

Named Arguments
--config-file

command config file (.yaml)

-v, --volume

A mount specification in the form of <host path>:<container path>. The given path on the host machine will be mounted under the given path in the command container.

Default: []

-c, --context

The filepath to a directory that contains the set of files used to execute the command. All files under this directory will be packaged, maintaining the existing directory structure. The total byte contents of the directory must not exceed 96 MB. By default, the context directory will be empty.

--config

Additional configuration arguments for setting up a command. Arguments should be specified as key=value. Nested configuration keys can be specified by dot notation, e.g., resources.slots=4. List values can be specified by comma-separated values.

Default: []

-p, --passphrase

passphrase to encrypt the shell private key

Default: False

--template

name of template to apply to the shell configuration

-d, --detach

run in the background and print the ID

Default: False

--show-ssh-command

show ssh command (e.g. for use in IDE) when starting the shell

Default: False

slot (s)

manage slots

det slot [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, disable, enable, list

Sub-commands:
help

show help for this command

det slot help [-h]
disable

disable slot on agent

det slot disable [-h] agent_id slot_id
Positional Arguments
agent_id

agent ID

slot_id

slot ID

enable

enable slot on agent

det slot enable [-h] agent_id slot_id
Positional Arguments
agent_id

agent ID

slot_id

slot ID

list

list slots in cluster

det slot list [-h] [--csv | --json]
Named Arguments
--csv

print as CSV

Default: False

--json

print as JSON

Default: False

task

manage tasks (commands, experiments, notebooks, shells, tensorboards)

det task [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, list

Sub-commands:
help

show help for this command

det task help [-h]
list

list tasks in cluster

det task list [-h] [--csv]
Named Arguments
--csv

print as CSV

Default: False

template (tpl)

manage config templates

det template [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, describe, list, ls, remove, rm, set

Sub-commands:
help

show help for this command

det template help [-h]
describe

describe config template

det template describe [-h] template_name
Positional Arguments
template_name

template name

list (ls)

list config templates

det template list [-h] [-d]
Named Arguments
-d, --details

show the configs of the templates

Default: False

remove (rm)

remove config template

det template remove [-h] template_name
Positional Arguments
template_name

template name

set

set config template

det template set [-h] template_name template_file
Positional Arguments
template_name

template name

template_file

config template file (.yaml)

tensorboard

manage TensorBoard instances

det tensorboard [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, config, kill, list, ls, logs, open, start

Sub-commands:
help

show help for this command

det tensorboard help [-h]
config

display TensorBoard config

det tensorboard config [-h] id
Positional Arguments
id

TensorBoard ID

kill

kill TensorBoard instance

det tensorboard kill [-h] [-f] tensorboard_id [tensorboard_id ...]
Positional Arguments
tensorboard_id

TensorBoard ID

Named Arguments
-f, --force

ignore errors

Default: False

list (ls)

list TensorBoard instances

det tensorboard list [-h] [-q] [--all]
Named Arguments
-q, --quiet

only display the IDs

Default: False

--all, -a

show all TensorBoards (including other users’)

Default: False

logs

fetch TensorBoard instance logs

det tensorboard logs [-h] [-f] [--tail TAIL] tensorboard_id
Positional Arguments
tensorboard_id

TensorBoard ID

Named Arguments
-f, --follow

follow the logs of a TensorBoard instance, similar to tail -f

Default: False

--tail

number of lines to show, counting from the end of the log

Default: 200

open

open existing TensorBoard instance

det tensorboard open [-h] tensorboard_id
Positional Arguments
tensorboard_id

TensorBoard ID

start

start new TensorBoard instance

det tensorboard start [-h] [--config-file CONFIG_FILE]
                      [-t TRIAL_IDS [TRIAL_IDS ...]] [--no-browser]
                      [-c CONTEXT] [-d]
                      [experiment_ids [experiment_ids ...]]
Positional Arguments
experiment_ids

experiment IDs to load into TensorBoard. At most 100 trials from the specified experiment will be loaded into TensorBoard. If the experiment has more trials, the 100 best-performing trials will be used.

Named Arguments
--config-file

command config file (.yaml)

-t, --trial-ids

trial IDs to load into TensorBoard; at most 100 trials are allowed per TensorBoard instance

--no-browser

don’t open TensorBoard in a browser after startup

Default: False

-c, --context

The filepath to a directory that contains the set of files used to execute the command. All files under this directory will be packaged, maintaining the existing directory structure. The total byte contents of the directory must not exceed 96 MB. By default, the context directory will be empty.

-d, --detach

run in the background and print the ID

Default: False

trial (t)

manage trials

det trial [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, describe, download, kill, logs

Sub-commands:
help

show help for this command

det trial help [-h]
describe

describe trial

det trial describe [-h] [--metrics] [--csv | --json] trial_id
Positional Arguments
trial_id

trial ID

Named Arguments
--metrics

display full metrics

Default: False

--csv

print as CSV

Default: False

--json

print JSON

Default: False

download

download checkpoint for trial

det trial download [-h] (--best | --latest | --uuid UUID) [-o OUTPUT_DIR]
                   [--sort-by SORT_BY] [--smaller-is-better SMALLER_IS_BETTER]
                   [-q]
                   trial_id
Positional Arguments
trial_id

trial ID

Named Arguments
--best

download the checkpoint with the best validation metric

Default: False

--latest

download the most recent checkpoint

Default: False

--uuid

download a checkpoint by specifying its UUID

-o, --output-dir

Desired output directory for the checkpoint

--sort-by

The name of the validation metric to sort on. This argument is only used with –best. If –best is passed without –sort-by, the experiment’s searcher metric is assumed. If this argument is specified, –smaller-is-better must also be specified.

--smaller-is-better

The sort order for metrics when using –best with –sort-by. For example, ‘accuracy’ would require passing ‘–smaller-is-better false’. If –sort-by is specified, this argument must be specified.

-q, --quiet

only print the path to the checkpoint

Default: False

kill

forcibly terminate a trial

det trial kill [-h] trial_id
Positional Arguments
trial_id

trial ID

logs

fetch trial logs

det trial logs [-h] [-f] [--head HEAD | --tail TAIL] [--agent-id AGENT_IDS]
               [--container-id CONTAINER_IDS] [--rank-id RANK_IDS]
               [--timestamp-before TIMESTAMP_BEFORE]
               [--timestamp-after TIMESTAMP_AFTER] [--level LEVEL]
               [--source SOURCES] [--stdtype STDTYPES]
               trial_id
Positional Arguments
trial_id

trial ID

Named Arguments
-f, --follow

follow the logs of a running trial, similar to tail -f

Default: False

--head

number of lines to show, counting from the beginning of the log (default is all)

--tail

number of lines to show, counting from the end of the log (default is all)

--agent-id

agents to show logs from (repeat for multiple values)

--container-id

containers to show logs from (repeat for multiple values)

--rank-id

containers to show logs from (repeat for multiple values)

--timestamp-before

show logs only from before (RFC 3339 format)

--timestamp-after

show logs only from after (RFC 3339 format)

--level

show logs with this level or higher (TRACE, DEBUG, INFO, WARNING, ERROR, CRITICAL)

--source

sources to show logs from (repeat for multiple values)

--stdtype

output stream to show logs from (repeat for multiple values)

user (u)

manage users

det user [-h] subcommand ...
Positional Arguments
subcommand

Possible choices: help, activate, change-password, create, deactivate, link-with-agent-user, list, login, logout, rename, whoami

Sub-commands:
help

show help for this command

det user help [-h]
activate

activate user

det user activate [-h] username
Positional Arguments
username

name of user to activate

change-password

change password for user

det user change-password [-h] [target_user]
Positional Arguments
target_user

name of user to change password of

create

create user

det user create [-h] [--admin] username
Positional Arguments
username

name of new user

Named Arguments
--admin

give new user admin rights

Default: False

deactivate

deactivate user

det user deactivate [-h] username
Positional Arguments
username

name of user to deactivate

list

list users

det user list [-h]
login

log in user

det user login [-h] [username]
Positional Arguments
username

name of user to log in as

logout

log out user

det user logout [-h]
rename

change username for user

det user rename [-h] target_user new_username
Positional Arguments
target_user

name of user whose username should be changed

new_username

new username for target_user

whoami

print the active user

det user whoami [-h]

version

show version information

det version [-h]

Environment Variables

  • DET_MASTER: The network address of the master of the Determined installation. The value can be overridden using the -m flag.

Examples

  • det e, det experiment, det experiment list: Show information about experiments in the cluster.

  • det -m 1.2.3.4 e, DET_MASTER=1.2.3.4 det e: Show information about experiments in the cluster at the network address 1.2.3.4.

  • det t logs -f 289: Show the existing logs for trial 289 and continue showing new logs as they come in.

  • det e label add 17 foobar: Add the label “foobar” to experiment 17.

  • det e describe 493 --metrics --csv: Display information about experiment 493, including full metrics information, in CSV format.

  • det e create -f --paused const.yaml .: Create an experiment with the configuration file const.yaml and the code contained in the current directory. The experiment will be created in a paused state (that is, it will not be scheduled on the cluster until it is activated).

  • det e set max-slots 85 4: Ensure that experiment 85 does not take up more than 4 slots in the cluster.

  • det u create --admin hoid: Create a new user named “hoid” with admin privileges.

  • det version: Show detailed information about the CLI and master. Note that this command does not take both an object and an action.