Dynamic Agents on GCP¶
This document describes how to install, configure, and upgrade a deployment of Determined with dynamic agents on GCP. See Elastic Infrastructure for an overview of the elastic infrastructure in Determined.
Compute Engine Project¶
The Determined master and the Determined agents are intended to run in the same project.
When using dynamic agents on GCP, Determined identifies the Compute Engine instances that it is managing using a configurable instance label (see Cluster Configuration for details). Administrators should be careful to ensure that this label is not used by other Compute Engine instances that are launched outside of Determined; if that assumption is violated, unexpected behavior may occur.
Compute Engine Images¶
The Determined master node will run on a custom image that will be shared with you by Determined AI.
Determined agent nodes will run on a custom image that will be shared with you by Determined AI.
Compute Engine Machine Types¶
The Determined master node should be deployed on a Compute Engine instance with >= 2 CPUs (Intel Broadwell or later), 4GB of RAM, and 100GB of disk storage. This would be a Compute Engine
n1-standard-2or more powerful.
GCP API Access¶
The Determined master needs to run as a service account that has the permissions to manage Compute Engine instances. There are two options:
Create a particular service account with the
Compute Adminrole. Then set the Determined master to use this account. See Compute Engine IAM roles for more details on how to configure the service account.
In order for the Determined agent to be associated with a service account, the Determined master needs to have access to service accounts. Please ensure the service account of the Determined master has the
Service Account Userrole.
In order for the Determined agent to use a shared VPC, the service account that the master runs with needs to have the
Compute Network Userrole.
Use the default service account and add the
Compute Engine: Read Writescope.
Optionally, the Determined agent may be associated with a service account.
Access scopes are the legacy method of specifying permissions for your instance. A best practice is to set the full cloud-platform access scope on the instance, then securely limit the service account’s API access with Cloud IAM roles. See Access Scopes for details.
The Determined Cluster is configured with
master.yaml file located
/usr/local/determined/etc on the Determined master instance.
Below you’ll find an example configuration. See
Cluster Configuration for details.
provisioner: master_url: <scheme://host:port> startup_script: <startup script> container_startup_script: <container startup script> agent_docker_network: host agent_docker_image: <agent docker image> max_idle_agent_period: 5m max_agent_starting_period: 5m provider: gcp base_config: <instance resource base configuration> project: <project ID> zone: <zone> boot_disk_size: 200 boot_disk_source_image: projects/<project-id>/global/images/<image-name> label_key: <label key for agent discovery> label_value: <label value for agent discovery> name_prefix: <name prefix> network_interface: network: projects/<project>/global/networks/<network> subnetwork: projects/<project>/regions/<region>/subnetworks/<subnetwork> external_ip: false network_tags: ["<tag1>", "<tag2>"] service_account: email: "<service account email>" scopes: ["https://www.googleapis.com/auth/cloud-platform"] instance_type: machine_type: n1-standard-32 gpu_type: nvidia-tesla-v100 gpu_num: 4 preemptible: false max_instances: 5 operation_timeout_period: 5m
Attaching A Disk To Each Dynamic Agent¶
If your input data set is on a persistent disk, you can attach that disk to each dynamic agent by using the base instance configuration and preparing commands. The following is an example configuration. See REST Resource: instances for the full list of configuration options supported by GCP. See Formatting and mounting a zonal persistent disk for more examples of formatting or mounting disks in GCP.
Here is an example master configuration to attach and mount a second disk to each dynamic agent.
provisioner: startup_script: | lsblk mkdir -p /mnt/disks/second mount -o discard,defaults /dev/sdb1 /mnt/disks/second lsblk provider: gcp base_config: disks: - mode: READ_ONLY boot: false source: zones/<zone>/disks/<the name of the existing disk> autoDelete: false boot_disk_size: 200 boot_disk_source_image: projects/<project>/global/images/<image name>
If a specific non-root user needs to access the disk, please run the tasks linked with the POSIX UID/GID of the user (See Running tasks as particular agent users for details.) and grant access to the corresponding UID/GID.
You can use the following command to validate if Determined tasks can read from the attached disk.
cat > command.yaml << EOF bind_mounts: - host_path: /mnt/disks/second container_path: /second EOF # Test attached read-only disk. det command run --config-file command.yaml ls -l /second
Securely Pulling Images from GCR¶
If you have expensive operations to perform at startup, it can be useful to add custom layers to the task images Determined provides. If you have store these images in a secure registry, such as GCR, you can pull these images securely by using existing tooling like docker-credential-gcr.
Here is an example master configuration of how to allow the agent to inherit the permissions of the service account associated with a GCE instance, for accessing GCR.
provisioner: container_startup_script: | export HOME=/root apt-get update && apt-get install -y curl docker.io curl -fsSL "https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v1.5.0/docker-credential-gcr_linux_amd64-1.5.0.tar.gz" \ | tar xz --to-stdout > /usr/bin/docker-credential-gcr && chmod +x /usr/bin/docker-credential-gcr docker-credential-gcr configure-docker
This is an example of an operation that requires use of
container_startup_script. Because docker credential helpers alter
the docker client configuration to depend on the helper binary by
name, it must be installed and configured in the container.
These instructions describe how to install Determined for the first time; for directions on how to upgrade an existing Determined installation, see the Upgrades section below.
Ensure that you are using the most up-to-date Determined images. Keep the image IDs handy as we will need them later.
To install the master, we will launch an instance from the Determined master image.
Let’s start by navigating to the Compute Engine Dashboard of the GCP Console. Click “Create Instance” and follow the instructions below:
Choose Machine Type: we recommend a
n1-standard-2or more powerful.
Configure Boot Disk:
Choose Boot Disk Image: find the Determined master image in “Images” and click “Select”.
Set Boot Disk Size: set
Sizeto be at least 100GB. If you have a previous Determined installation that you are upgrading, you want to use the snapshot or existing disk. This disk will be used to store all your experiment metadata and checkpoints.
Configure Identity and API access: choose the
service accountaccording to GCP API Access.
Configure Firewalls: choose or create a security group according to these Network Requirements. Check off
Allow HTTP traffic.
Review and launch the instance.
SSH into the Determined master and edit the config at
/usr/local/determined/etc/master.yamlaccording to the guide on Cluster Configuration.
Start the Determined master by entering
make -C /usr/local/determined enable-masterinto the terminal.