Shortcuts

Dynamic Agents on AWS

This document describes how to install, configure, and upgrade a deployment of Determined with dynamic agents on AWS. See Elastic Infrastructure for an overview of using elastic infrastructure in Determined.

Determined is able to launch dynamic agents as spot instances, which can be much cheaper than using standard on-demand instances. For more details on spot instances, see AWS Spot Instances.

System Requirements

EC2 Instance Tags

An important assumption of Determined with dynamic agents is that any EC2 instances with the configured tag_key:tag_value pair are managed by the Determined master (See Cluster Configuration). This pair should be unique to your Determined installation. If it is not, Determined may inadvertently manage your non-Determined EC2 instances.

If using spot instances, Determined also assumes that any EC2 spot instance requests with the configured tag_key:tag_value pair are managed by the Determined master.

EC2 AMIs

  • The Determined master node will run on a custom AMI that will be shared with you by Determined AI.

  • Determined agent nodes will run on a custom AMI that will be shared with you by Determined AI.

EC2 Instance Types

  • The Determined master node should be deployed on an EC2 instance supporting >= 2 CPUs (Intel Broadwell or later), 4GB of RAM, and 100GB of disk storage. This corresponds to an EC2 t2.medium instance or better.

  • All Determined agent nodes must be the same AWS instance type; any G4, P2, or P3 instance type is supported. This can be configured in the Cluster Configuration.

Master IAM Role

The Determined master needs to have an IAM role with the following permissions:

  • ec2:CreateTags: used to tag the Determined agent instances that the Determined master provisions. These tags are configured by the aws-cluster-configuration.

  • ec2:DescribeInstances: used to find active Determined agent instances based on tags.

  • ec2:RunInstances: used to provision Determined agent instances.

  • ec2:TerminateInstances: used to terminate idle Determined agent instances.

If using spot instances, the master also needs the following permissions:

  • ec2:RequestSpotInstances: used to provision Determined agent instances as spot instances.

  • ec2:CancelSpotInstanceRequests: used to adjust the number of spot instance requests to match the number of instances needed for the current workloads.

  • ec2:DescribeSpotInstanceRequests: used to find open spot instance requests that, once fulfilled, will create Determined agent spot instances.

An example IAM policy with the appropriate permissions is below:

{
  "Version": "2012-10-17",
  "Statement": [
     {
       "Sid": "VisualEditor0",
       "Effect": "Allow",
       "Action": [
         "ec2:DescribeInstances",
         "ec2:TerminateInstances",
         "ec2:CreateTags",
         "ec2:RunInstances",
         "ec2:CancelSpotInstanceRequests",
         "ec2:RequestSpotInstances",
         "ec2:DescribeSpotInstanceRequests",
       ],
       "Resource": "*"
     }
  ]
}

If you need to attach an instance profile to the agent (e.g., iam_instance_profile_arn is set in the Cluster Configuration), make sure to add PassRole policy to the master role with Resource set to the desired agent role. For example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "<arn::agent-role>"
    }
  ]
}

See Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances for details.

Network Requirements

See Network Requirements for details.

Cluster Configuration

The Determined Cluster is configured with master.yaml file located at /usr/local/determined/etc/ on the Determined master instance. We need to configure AWS dynamic agents in each resource pool. See Cluster Configuration for details.

Installation

These instructions describe how to install Determined for the first time; for directions on how to upgrade an existing Determined installation, see the Upgrades section below.

Ensure that you are using the most up-to-date Determined AMIs. Keep the AMI IDs handy as we will need them later (e.g., ami-0f4677bfc3161edc8).

Master

To install the master, we will launch an instance from the Determined master AMI.

Let’s start by navigating to the EC2 Dashboard of the AWS Console. Click “Launch Instance” and follow the instructions below:

  1. Choose AMI: find the Determined master AMI in “My AMIs” and click “Select”.

  2. Choose Instance Type: we recommend a t2.medium or more powerful.

  3. Configure Instance: choose the IAM role according to Master IAM Role.

  4. Add Storage: click Add New Volume and add an EBS volume of at least 100GB. If you have a previous Determined installation that you are upgrading, you want to use the attach the same EBS volume as the previous installation. This volume will be used to store all your experiment metadata and checkpoints.

  5. Configure Security Group: choose or create a security group according to Network Requirements.

  6. Review and launch the instance.

  7. SSH into the Determined master and edit the config at /usr/local/determined/etc/master.yaml according to the guide on Cluster Configuration.

  8. Start the Determined master by entering make -C /usr/local/determined enable-master into the terminal.

Agent

There is no installation needed for the agent. The Determined master will dynamically launch Determined agent instances based on the Cluster Configuration.

Upgrades

Upgrading an existing Determined installation with dynamic agents on AWS requires the same steps as an installation without dynamic agents. See Upgrades.

Monitoring

Both the Determined master and agent AMIs are configured to forward system journald logs and basic GPU metrics to AWS CloudWatch when their instances have the appropriate IAM permissions. These logs and metrics can be helpful for diagnosing infrastructure issues when using Dynamic Agents on AWS.

CloudWatch Logging

An instance needs the following permissions to upload logs to CloudWatch:

  • logs:CreateLogStream

  • logs:PutLogEvents

  • logs:DescribeLogStreams

Instances will upload their logs to the log group /determined/determined/journald. This log group must be created in advance before any logs can be stored.

An example IAM policy with the appropriate permissions is below:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
      ],
      "Resource": [
        "arn:aws:logs:*:*:log-group:/determined/determined/journald",
        "arn:aws:logs:*:*:log-group:/determined/determined/journald:log-stream:*"
      ]
    }
  ]
}

CloudWatch Metrics

An instance needs the following permissions to upload logs to CloudWatch:

  • cloudwatch:PutMetricData

Instances will upload their metrics to namespace Determined.

An example IAM policy with the appropriate permissions is below.

{
  "Version": "2012-10-17",
  "Statement": [
    {
     "Action": [
       "cloudwatch:PutMetricData"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}