Transport Layer Security

Transport Layer Security (TLS) is a protocol for secure network communication. TLS prevents the data being transmitted from being modified or read while it is in transit and allows clients to verify the identity of the server (in this case, the Determined master). Determined can be configured to use TLS for all connections made to the master. That means that all CLI and WebUI connections will be secured by TLS, as well as connections from agents and tasks to the master. Communication between agents that occur as part of distributed training will not use TLS, nor will proxied connections from the master to a TensorBoard or notebook instance.

Configuring TLS

Master

In order to configure the master to use TLS, set the security.tls.cert and security.tls.key options to paths to a TLS certificate file and key file.

When TLS is in use, the master will listen on TCP port 8443 by default, rather than 8080.

Note

If the master’s certificate is not signed by a well-known CA, then the configured certificate file must contain a full certificate chain that goes all the way to a root certificate.

Agents

When the Determined master is using TLS, set the security.tls.enabled agent configuration option to true. If the master’s certificate is signed by a well-known CA, then no other TLS-specific configuration is necessary. Otherwise, for the best security, place the master’s certificate file somewhere accessible to the agent and set the agent’s security.tls.master_cert option to the path to that file. For a more convenient but less secure setup, instead set the security.tls.skip_verify option to true. With the latter configuration, the agent will be unable to verify the identity of the master, but the data sent over the connection will still be protected by TLS.

When dynamic agents and TLS are both in use, the dynamic agents that the master creates will automatically be configured to connect securely to the master over TLS.

CLI

In order to use TLS, the CLI must be configured with a master address starting with https:// using either the -m flag or DET_MASTER environment variable.

If the master’s certificate is signed by a well-known CA, then the connection should proceed immediately. If not, the CLI will indicate on the first connection that the master is presenting an untrusted certificate and display a hash of the certificate. You may wish to confirm the hash with your system administrator; in any case, if you confirm the connection to the master, the certificate will be stored on the computer where the CLI is being run and future connections to the master will be made without confirmation.

Tasks

Once the master and agent are configured to use TLS, no further configuration is required for tasks that are run in the cluster. In shells and notebooks, the Determined Python libraries will automatically make connections to the master using TLS with the appropriate certificate.