Command-line Interface¶
The Determined command-line interface (CLI) is installed under the name
det
. Most uses of it require arguments specifying a type of object
to act on and the action to perform, followed by any additional
arguments or options. For example,
det experiment config 17
displays the configuration for experiment 17. Some commands have an additional level of nesting:
det experiment label add 17 foobar
adds the label “foobar” to experiment 17.
The names of some objects, actions, and options may be abbreviated; alternative names are shown in the full reference in parentheses. For example, these two commands are equivalent:
det -u admin experiment list
det --user admin experiment list
as are these two:
det e config 17
det experiment config 17
Additionally, when list
is a valid action, it is the default when no
action is specified, so the following pairs of commands are also
equivalent:
det e
det experiment list
det s
det slot list
Providing -h
or --help
as an argument anywhere will cause the
CLI to exit after printing help text for the object or action specified
up to that point.
Full Listing¶
Determined command-line client
usage: det [-h] [-u username] [-m address] [-v] command ...
Positional Arguments¶
- command
Possible choices: help, agent, a, command, cmd, checkpoint, c, experiment, e, master, m, model, notebook, preview-search, shell, slot, s, task, template, tpl, tensorboard, trial, t, user, u, version
Named Arguments¶
- -u, --user
run as the given user
- -m, --master
master address
Default: “localhost:8080”
- -v, --version
print CLI version and exit
Sub-commands:¶
agent (a)¶
manage agents
det agent [-h] subcommand ...
Positional Arguments¶
- subcommand
Possible choices: help, disable, enable, list
Sub-commands:¶
disable¶
disable agent
det agent disable [-h] [--all] [agent_id]
- agent_id
agent ID
- --all
disable all agents
Default: False
command (cmd)¶
manage commands
det command [-h] subcommand ...
Positional Arguments¶
- subcommand
Possible choices: help, config, kill, list, ls, logs, run
Sub-commands:¶
kill¶
forcibly terminate a command
det command kill [-h] [-f] command_id [command_id ...]
- command_id
command ID
- -f, --force
ignore errors
Default: False
list (ls)¶
list commands
det command list [-h] [-q] [--all]
- -q, --quiet
only display the IDs
Default: False
- --all, -a
show all commands (including other users’)
Default: False
logs¶
fetch command logs
det command logs [-h] [-f] [--tail TAIL] command_id
- command_id
command ID
- -f, --follow
follow the logs of a command, similar to tail -f
Default: False
- --tail
number of lines to show, counting from the end of the log
Default: 200
run¶
create command
det command run [-h] [--config-file CONFIG_FILE] [-v VOLUME] [-c CONTEXT]
[--config CONFIG] [--template TEMPLATE] [-d]
...
- entrypoint
entrypoint command and arguments to execute
- --config-file
command config file (.yaml)
- -v, --volume
A mount specification in the form of <host path>:<container path>. The given path on the host machine will be mounted under the given path in the command container.
Default: []
- -c, --context
The filepath to a directory that contains the set of files used to execute the command. All files under this directory will be packaged, maintaining the existing directory structure. The total byte contents of the directory must not exceed 96 MB. By default, the context directory will be empty.
- --config
Additional configuration arguments for setting up a command. Arguments should be specified as key=value. Nested configuration keys can be specified by dot notation, e.g., resources.slots=4. List values can be specified by comma-separated values.
Default: []
- --template
name of template to apply to the command configuration
- -d, --detach
run in the background and print the ID
Default: False
checkpoint (c)¶
manage checkpoints
det checkpoint [-h] subcommand ...
Positional Arguments¶
- subcommand
Possible choices: help, describe, download
Sub-commands:¶
describe¶
describe checkpoint
det checkpoint describe [-h] uuid
- uuid
checkpoint uuid to describe
download¶
download checkpoint from persistent storage
det checkpoint download [-h] [-o OUTPUT_DIR] [-q] uuid
- uuid
Download a checkpoint by specifying its UUID.
- -o, --output-dir
Desired output directory for the checkpoint.
- -q, --quiet
Only print the path to the checkpoint.
Default: False
experiment (e)¶
manage experiments
det experiment [-h] subcommand ...
Positional Arguments¶
- subcommand
Possible choices: help, activate, archive, cancel, config, create, describe, download, download-model-def, kill, label, list, list-checkpoints, lc, list-trials, lt, pause, set, unarchive, wait
Sub-commands:¶
activate¶
activate experiment
det experiment activate [-h] experiment_id
- experiment_id
experiment ID to activate
archive¶
archive experiment
det experiment archive [-h] experiment_id
- experiment_id
experiment ID to archive
cancel¶
cancel experiment
det experiment cancel [-h] experiment_id
- experiment_id
experiment ID to cancel
config¶
display experiment config
det experiment config [-h] experiment_id
- experiment_id
experiment ID
create¶
create experiment
det experiment create [-h] [-g] [--local] [--template TEMPLATE]
[-f | --paused | -t]
config_file model_def
- config_file
experiment config file (.yaml)
- model_def
file or directory containing model definition
- -g, --git
Associate git metadata with this experiment. This flag assumes that git is installed, a .git repository exists in the model definition directory, and that the git working tree of that repository is empty.
Default: False
- --local
Create the experiment in local mode instead of submitting it to the cluster. For more information, see documentation on det.experimental.create()
Default: False
- --template
name of template to apply to the experiment configuration
- -f, --follow-first-trial
follow the logs of the first trial that is created
Default: False
- --paused
do not activate the experiment
Default: False
- -t, --test-mode
Test the experiment configuration and model definition by creating and scheduling a very small experiment. This command will verify that a training workload and validation workload run successfully and that checkpoints can be saved. The test experiment will be archived on creation.
Default: False
describe¶
describe experiment
det experiment describe [-h] [--metrics] [--csv | --json | --outdir OUTDIR]
experiment_ids
- experiment_ids
comma-separated list of experiment IDs to describe
- --metrics
display full metrics
Default: False
- --csv
print as CSV
Default: False
- --json
print as JSON
Default: False
- --outdir
directory to save output
download¶
download checkpoints for an experiment
det experiment download [-h] [-o OUTPUT_DIR] [--top-n TOP_N]
[--sort-by SORT_BY]
[--smaller-is-better SMALLER_IS_BETTER] [-q]
experiment_id
- experiment_id
experiment ID to cancel
- -o, --output-dir
Desired top level directory for the checkpoints. Checkpoints will be downloaded to <output_dir>/<checkpoint_uuid>/<checkpoint_files>.
Default: “checkpoints”
- --top-n
The number of checkpoints to download for the experiment. The checkpoints are sorted by validation metric as defined by –sort-by and –smaller-is-better.This command will select the best N checkpoints from the top performing N trials of the experiment.
Default: 1
- --sort-by
The name of the validation metric to sort on. Without –sort-by, the experiment’s searcher metric is assumed. If this argument is specified, –smaller-is-better must also be specified.
- --smaller-is-better
The sort order for metrics when using –sort-by. For example, ‘accuracy’ would require passing ‘–smaller-is-better false’. If –sort-by is specified, this argument must be specified.
- -q, --quiet
Only print the paths to the checkpoints.
Default: False
download-model-def¶
download model definition
det experiment download-model-def [-h] [--output-dir OUTPUT_DIR] experiment_id
- experiment_id
experiment ID
- --output-dir
output directory
Default: .
kill¶
kill experiment
det experiment kill [-h] experiment_id
- experiment_id
experiment ID
label¶
manage experiment labels
det experiment label [-h] subsubcommand ...
- subsubcommand
Possible choices: help, add, remove
list¶
list experiments
det experiment list [-h] [--all] [--csv]
- --all, -a
show all experiments (including archived and other users’)
Default: False
- --csv
print as CSV
Default: False
list-checkpoints (lc)¶
list checkpoints of experiment
det experiment list-checkpoints [-h] [--best BEST] [--csv] experiment_id
- experiment_id
experiment ID
- --best
Return the best N checkpoints for this experiment. If this flag is used, only checkpoints with an associated validation metric will be considered.
- --csv
print as CSV
Default: False
list-trials (lt)¶
list trials of experiment
det experiment list-trials [-h] [--csv] experiment_id
- experiment_id
experiment ID
- --csv
print as CSV
Default: False
pause¶
pause experiment
det experiment pause [-h] experiment_id
- experiment_id
experiment ID to pause
set¶
set experiment attributes
det experiment set [-h] subsubcommand ...
- subsubcommand
Possible choices: help, description, gc-policy, max-slots, weight
set experiment description
det experiment set description [-h] experiment_id description
- experiment_id
experiment ID to modify
- description
experiment description
set experiment GC policy and run GC
det experiment set gc-policy [-h] --save-experiment-best SAVE_EXPERIMENT_BEST
--save-trial-best SAVE_TRIAL_BEST
--save-trial-latest SAVE_TRIAL_LATEST [--yes]
experiment_id
- experiment_id
experiment ID to modify
- --save-experiment-best
number of best checkpoints per experiment to save
- --save-trial-best
number of best checkpoints per trial to save
- --save-trial-latest
number of latest checkpoints per trial to save
- --yes
automatically answer yes to prompts
Default: False
master¶
manage master
det master [-h] subcommand ...
Positional Arguments¶
- subcommand
Possible choices: help, config, logs
Sub-commands:¶
m (model)¶
Undocumented
det m [-h] subcommand ...
Positional Arguments¶
- subcommand
Possible choices: help, create, describe, list, list-versions, register-version
Sub-commands:¶
create¶
create model
det m create [-h] [--description DESCRIPTION] [--json] name
- name
unique name of the model
- --description
description of the model
- --json
print as JSON
Default: False
describe¶
describe model
det m describe [-h] [--json] [--version VERSION] name
- name
model to describe
- --json
print as JSON
Default: False
- --version
model version information to include in output
Default: 0
list¶
list all models in the registry
det m list [-h] [--sort-by {name,description,creation_time,last_updated_time}]
[--order-by {asc,desc}] [--json]
- --sort-by
Possible choices: name, description, creation_time, last_updated_time
sort models by the given field
Default: “last_updated_time”
- --order-by
Possible choices: asc, desc
order models in either ascending or descending order
Default: “asc”
- --json
print as JSON
Default: False
notebook¶
manage notebooks
det notebook [-h] subcommand ...
Positional Arguments¶
- subcommand
Possible choices: help, config, kill, list, ls, logs, open, start
Sub-commands:¶
kill¶
kill a notebook
det notebook kill [-h] [-f] notebook_id [notebook_id ...]
- notebook_id
notebook ID
- -f, --force
ignore errors
Default: False
list (ls)¶
list notebooks
det notebook list [-h] [-q] [--all]
- -q, --quiet
only display the IDs
Default: False
- --all, -a
show all notebooks (including other users’)
Default: False
logs¶
fetch notebook logs
det notebook logs [-h] [-f] [--tail TAIL] notebook_id
- notebook_id
notebook ID
- -f, --follow
follow the logs of a notebook, similar to tail -f
Default: False
- --tail
number of lines to show, counting from the end of the log
Default: 200
open¶
open an existing notebook
det notebook open [-h] notebook_id
- notebook_id
notebook ID
start¶
start a new notebook
det notebook start [-h] [--config-file CONFIG_FILE] [-v VOLUME] [-c CONTEXT]
[--config CONFIG] [--template TEMPLATE] [--no-browser] [-d]
- --config-file
command config file (.yaml)
- -v, --volume
A mount specification in the form of <host path>:<container path>. The given path on the host machine will be mounted under the given path in the command container.
Default: []
- -c, --context
The filepath to a directory that contains the set of files used to execute the command. All files under this directory will be packaged, maintaining the existing directory structure. The total byte contents of the directory must not exceed 96 MB. By default, the context directory will be empty.
- --config
Additional configuration arguments for setting up a command. Arguments should be specified as key=value. Nested configuration keys can be specified by dot notation, e.g., resources.slots=4. List values can be specified by comma-separated values.
Default: []
- --template
name of template to apply to the notebook configuration
- --no-browser
don’t open the notebook in a browser after startup
Default: False
- -d, --detach
run in the background and print the ID
Default: False
preview-search¶
preview search
det preview-search [-h] config_file
Positional Arguments¶
- config_file
experiment config file (.yaml)
shell¶
manage shells
det shell [-h] subcommand ...
Positional Arguments¶
- subcommand
Possible choices: help, config, kill, list, logs, open, start
Sub-commands:¶
kill¶
kill a shell
det shell kill [-h] [-f] shell_id [shell_id ...]
- shell_id
shell ID
- -f, --force
ignore errors
Default: False
list¶
list shells
det shell list [-h] [-q] [--all]
- -q, --quiet
only display the IDs
Default: False
- --all, -a
show all shells (including other users’)
Default: False
logs¶
fetch shell logs
det shell logs [-h] [-f] [--tail TAIL] shell_id
- shell_id
shell ID
- -f, --follow
follow the logs of a shell, similar to tail -f
Default: False
- --tail
number of lines to show, counting from the end of the log
Default: 200
open¶
open an existing shell
det shell open [-h] shell_id [ssh_opts [ssh_opts ...]]
- shell_id
shell ID
- ssh_opts
additional SSH options when connecting to the shell
start¶
start a new shell
det shell start [-h] [--config-file CONFIG_FILE] [-v VOLUME] [-c CONTEXT]
[--config CONFIG] [-p] [--template TEMPLATE] [-d]
[ssh_opts [ssh_opts ...]]
- ssh_opts
additional SSH options when connecting to the shell
- --config-file
command config file (.yaml)
- -v, --volume
A mount specification in the form of <host path>:<container path>. The given path on the host machine will be mounted under the given path in the command container.
Default: []
- -c, --context
The filepath to a directory that contains the set of files used to execute the command. All files under this directory will be packaged, maintaining the existing directory structure. The total byte contents of the directory must not exceed 96 MB. By default, the context directory will be empty.
- --config
Additional configuration arguments for setting up a command. Arguments should be specified as key=value. Nested configuration keys can be specified by dot notation, e.g., resources.slots=4. List values can be specified by comma-separated values.
Default: []
- -p, --passphrase
passphrase to encrypt the shell private key
Default: False
- --template
name of template to apply to the shell configuration
- -d, --detach
run in the background and print the ID
Default: False
slot (s)¶
manage slots
det slot [-h] subcommand ...
Positional Arguments¶
- subcommand
Possible choices: help, disable, enable, list
task¶
manage tasks (commands, experiments, notebooks, shells, tensorboards)
det task [-h] subcommand ...
Positional Arguments¶
- subcommand
Possible choices: help, list
template (tpl)¶
manage config templates
det template [-h] subcommand ...
Positional Arguments¶
- subcommand
Possible choices: help, describe, list, ls, remove, rm, set
Sub-commands:¶
describe¶
describe config template
det template describe [-h] template_name
- template_name
template name
list (ls)¶
list config templates
det template list [-h] [-d]
- -d, --details
show the configs of the templates
Default: False
tensorboard¶
manage TensorBoard instances
det tensorboard [-h] subcommand ...
Positional Arguments¶
- subcommand
Possible choices: help, config, kill, list, ls, logs, open, start
Sub-commands:¶
config¶
display TensorBoard config
det tensorboard config [-h] tensorboard_id
- tensorboard_id
TensorBoard ID
kill¶
kill TensorBoard instance
det tensorboard kill [-h] [-f] tensorboard_id [tensorboard_id ...]
- tensorboard_id
TensorBoard ID
- -f, --force
ignore errors
Default: False
list (ls)¶
list TensorBoard instances
det tensorboard list [-h] [-q] [--all]
- -q, --quiet
only display the IDs
Default: False
- --all, -a
show all TensorBoards (including other users’)
Default: False
logs¶
fetch TensorBoard instance logs
det tensorboard logs [-h] [-f] [--tail TAIL] tensorboard_id
- tensorboard_id
TensorBoard ID
- -f, --follow
follow the logs of a TensorBoard instance, similar to tail -f
Default: False
- --tail
number of lines to show, counting from the end of the log
Default: 200
open¶
open existing TensorBoard instance
det tensorboard open [-h] tensorboard_id
- tensorboard_id
TensorBoard ID
start¶
start new TensorBoard instance
det tensorboard start [-h] [--config-file CONFIG_FILE]
[-t TRIAL_IDS [TRIAL_IDS ...]] [--no-browser]
[-c CONTEXT] [-d]
[experiment_ids [experiment_ids ...]]
- experiment_ids
experiment IDs to load into TensorBoard. At most 100 trials from the specified experiment will be loaded into TensorBoard. If the experiment has more trials, the 100 best-performing trials will be used.
- --config-file
command config file (.yaml)
- -t, --trial-ids
trial IDs to load into TensorBoard; at most 100 trials are allowed per TensorBoard instance
- --no-browser
don’t open TensorBoard in a browser after startup
Default: False
- -c, --context
The filepath to a directory that contains the set of files used to execute the command. All files under this directory will be packaged, maintaining the existing directory structure. The total byte contents of the directory must not exceed 96 MB. By default, the context directory will be empty.
- -d, --detach
run in the background and print the ID
Default: False
trial (t)¶
manage trials
det trial [-h] subcommand ...
Positional Arguments¶
- subcommand
Possible choices: help, describe, download, kill, logs
Sub-commands:¶
describe¶
describe trial
det trial describe [-h] [--metrics] [--csv | --json] trial_id
- trial_id
trial ID
- --metrics
display full metrics
Default: False
- --csv
print as CSV
Default: False
- --json
print JSON
Default: False
download¶
download checkpoint for trial
det trial download [-h] (--best | --latest | --uuid UUID) [-o OUTPUT_DIR]
[--sort-by SORT_BY] [--smaller-is-better SMALLER_IS_BETTER]
[-q]
trial_id
- trial_id
trial ID
- --best
download the checkpoint with the best validation metric
Default: False
- --latest
download the most recent checkpoint
Default: False
- --uuid
download a checkpoint by specifying its UUID
- -o, --output-dir
Desired output directory for the checkpoint
- --sort-by
The name of the validation metric to sort on. This argument is only used with –best. If –best is passed without –sort-by, the experiment’s searcher metric is assumed. If this argument is specified, –smaller-is-better must also be specified.
- --smaller-is-better
The sort order for metrics when using –best with –sort-by. For example, ‘accuracy’ would require passing ‘–smaller-is-better false’. If –sort-by is specified, this argument must be specified.
- -q, --quiet
only print the path to the checkpoint
Default: False
kill¶
forcibly terminate a trial
det trial kill [-h] trial_id
- trial_id
trial ID
logs¶
fetch trial logs
det trial logs [-h] [-f] [--head HEAD | --tail TAIL] [--agent-id AGENT_IDS]
[--container-id CONTAINER_IDS] [--rank-id RANK_IDS]
[--timestamp-before TIMESTAMP_BEFORE]
[--timestamp-after TIMESTAMP_AFTER] [--level LEVEL]
[--source SOURCES] [--stdtype STDTYPES]
trial_id
- trial_id
trial ID
- -f, --follow
follow the logs of a running trial, similar to tail -f
Default: False
- --head
number of lines to show, counting from the beginning of the log (default is all)
- --tail
number of lines to show, counting from the end of the log (default is all)
- --agent-id
agents to show logs from (repeat for multiple values)
- --container-id
containers to show logs from (repeat for multiple values)
- --rank-id
containers to show logs from (repeat for multiple values)
- --timestamp-before
show logs only from before (RFC 3339 format)
- --timestamp-after
show logs only from after (RFC 3339 format)
- --level
show logs with this level or higher (TRACE, DEBUG, INFO, WARNING, ERROR, CRITICAL)
- --source
sources to show logs from (repeat for multiple values)
- --stdtype
output stream to show logs from (repeat for multiple values)
user (u)¶
manage users
det user [-h] subcommand ...
Positional Arguments¶
- subcommand
Possible choices: help, activate, change-password, create, deactivate, link-with-agent-user, list, login, logout, rename, whoami
Sub-commands:¶
activate¶
activate user
det user activate [-h] username
- username
name of user to activate
change-password¶
change password for user
det user change-password [-h] [target_user]
- target_user
name of user to change password of
create¶
create user
det user create [-h] [--admin] username
- username
name of new user
- --admin
give new user admin rights
Default: False
deactivate¶
deactivate user
det user deactivate [-h] username
- username
name of user to deactivate
link-with-agent-user¶
link a user with UID/GID on agent
det user link-with-agent-user [-h] [--agent-uid AGENT_UID]
[--agent-user AGENT_USER]
[--agent-gid AGENT_GID]
[--agent-group AGENT_GROUP]
det_username
- det_username
name of Determined user to link
- --agent-uid
UID on the agent to run tasks as
- --agent-user
user on the agent to run tasks as
- --agent-gid
GID on agent to run tasks as
- --agent-group
group on the agent to run tasks as
login¶
log in user
det user login [-h] [username]
- username
name of user to log in as
Environment Variables¶
DET_MASTER
: The network address of the master of the Determined installation. The value can be overridden using the-m
flag.
Examples¶
det e
,det experiment
,det experiment list
: Show information about experiments in the cluster.det -m 1.2.3.4 e
,DET_MASTER=1.2.3.4 det e
: Show information about experiments in the cluster at the network address1.2.3.4
.det t logs -f 289
: Show the existing logs for trial 289 and continue showing new logs as they come in.det e label add 17 foobar
: Add the label “foobar” to experiment 17.det e describe 493 --metrics --csv
: Display information about experiment 493, including full metrics information, in CSV format.det e create -f --paused const.yaml .
: Create an experiment with the configuration fileconst.yaml
and the code contained in the current directory. The experiment will be created in a paused state (that is, it will not be scheduled on the cluster until it is activated).det e set max-slots 85 4
: Ensure that experiment 85 does not take up more than 4 slots in the cluster.det u create --admin hoid
: Create a new user named “hoid” with admin privileges.det version
: Show detailed information about the CLI and master. Note that this command does not take both an object and an action.