Dynamic Agents on AWS

This document describes how to install, configure, and upgrade a deployment of PEDL with Dynamic Agents on AWS. PEDL consists of several components:

  • a master that schedules workloads and stores metadata
  • one or more agents that run workloads, typically using GPUs

When running PEDL with Dynamic Agents, the PEDL master dynamically provisions and terminates EC2 instances to meet the needs of the cluster.

  • Provisioning new PEDL agents is quick: we make API calls to AWS to provision new instances within a few seconds of new tasks arriving. Within a few minutes new instances will have registered themselves with the PEDL master and start running tasks.
  • When PEDL agents become idle, we give them a five minute grace period before terminating the instances. This grace period provides for a short interval of time for the PEDL agent instance to receive new tasks.

The PEDL master and agents should typically be installed and configured by a system administrator. Each user of PEDL should also install a copy of the command-line tools, as described here.

These instructions describe how to install PEDL with Dynamic Agents on AWS.

System Requirements

EC2 Instance Tags

An important assumption of PEDL with Dynamic Agents is that any EC2 instances with the configured tag_key:tag_value pair are managed by the PEDL master (See configuration). If this pair is not unique to your PEDL installation, there will be unexpected behavior for your installation of PEDL and any EC2 instances with the configured tag_key:tag_value pair.

EC2 AMIs

  • The PEDL master node will run on a custom AMI that will be shared with you by Determined AI.

  • PEDL agent nodes will run on a custom AMI that will be shared with you by Determined AI.

EC2 Instance Types

  • The PEDL master node should be deployed on an EC2 instance supporting >= 2 CPUs (Intel Broadwell or later), 4GB of RAM, and 100GB of disk storage. This would be an EC2 t2.medium or more powerful.

  • Each PEDL agent node must be any of the P3 or P2 instances on AWS. This can be configured in the Cluster Configuration.

Master IAM Role

The PEDL master needs to have an IAM role with the following permissions:

  • ec2:CreateTags: used to tag the PEDL agent instances that the PEDL master provisions. These tags are configured by the Cluster Configuration.

  • ec2:DescribeInstances: used to find active PEDL agent instances based on tags.

  • ec2:RunInstances: used to provision PEDL agent instances.

  • ec2:TerminateInstances: used to terminate idle PEDL agent instances.

An example IAM policy with the appropriate permissions is below:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:TerminateInstances",
                "ec2:CreateTags",
                "ec2:RunInstances"
            ],
            "Resource": "*"
        }
    ]
}

Network Requirements

See Network Requirements for details.

Cluster Configuration

The PEDL Cluster is configured with master.yaml file located at /usr/local/pedl/etc/ on the PEDL master instance. Below you'll find an example configuration and an explanation for each field.

provisioner:
  master_url: <scheme://host:port>
  startup_script: <startup script>
  agent_docker_network: pedl
  max_idle_agent_period: 5m

  provider: aws
  region: <region>
  root_volume_size: 200
  image_id: <AMI id>
  tag_key: <tag key for agent discovery>
  tag_value: <tag value for agent discovery>
  instance_name: determined-ai-agent
  ssh_key_name: <ssh key name>
  network_interface:
    public_ip: true
    security_group_id: <security group id>
    subnet_id: <subnet id>
  instance_type: p3.8xlarge
  max_instances: 5
  • provisioner: top level field that contains the configuration needed for the PEDL master to provision the PEDL agent instances.

  • master_url: the full url of the master. A valid url is in the format of scheme://host:port. The scheme must be either http or https. If the master is deployed on EC2, rather than hardcoding the ip address, we advise you use one of the following to set the host as an alias: local-ipv4, public-ipv4, local-hostname, or public-hostname. Which one you should select is based on your network configuration. On master startup, we will replace the above alias host with its real value. Defaults to http as scheme, local ip address as host, and 8080 as port. (Optional)

  • startup_script: startup script for agents. This script will run right away when agent instances start up. For example, it can be used for formating and mounting a disk. Defaults to an empty string.

  • agent_docker_network: the Docker network to use for the PEDL agent and task containers. If this is set to "host", Docker host-mode networking will be used instead. The default value is "pedl".

  • max_idle_agent_period: length of the waiting period before terminating an idle agent instance. This string is a sequence of decimal numbers, each with optional fraction and a unit suffix, such as "30s", "1h", or "1m30s". Valid time units are "s", "m", "h". (Optional)

  • provider: the provider to provision instances with. To run dynamic agents on AWS, set it to be aws. (Required)

  • region: the region of the AWS resources used by PEDL. We advise setting this region to be the same region as the PEDL master for better network performance. Defaults to the same region as the master. (Optional)

  • root_volume_size: size of the root volume of the PEDL agent in GB. We recommend at least 100GB. Defaults to 200. (Optional)

  • image_id: the AMI ID of the PEDL agent that was shared with you. (Required)

  • tag_key: key for tagging the PEDL agent instances. Defaults to managed-by. (Optional)

  • tag_value: value for tagging the PEDL agent instances. Defaults to the master instance id if the master is on EC2 otherwise determined-ai-pedl. (Optional)

  • instance_name: name to set for the PEDL agent instances. Defaults to determined-ai-agent. (Optional)

  • ssh_key_name: the name of the ssh key registered with AWS for ssh key access to the agent instances. (Required)

  • network_interface: network interface to set for the PEDL agent instances. (Optional)

    • public_ip: flag to using public IP address for the PEDL agent instances. See Network Requirements for instruction on whether an external IP should be set. Defaults to false. (Optional)

    • security_group_id: the ID of the security group to run the PEDL agents as. This is the one you identified or created in Network Requirements . Defaults to the default security group of the specified VPC. (Optional)

    • subnet_id: the ID of the subnet to run the PEDL agents in. Defaults to the default subnet of the default VPC. (Optional)

  • max_instances: max number of PEDL agent instances. Defaults to 5. (Optional)

  • instance_type: type of instance for the PEDL agents. We only support P3 and P2 type instances. Defaults to p3.8xlarge. (Optional)

Installation

These instructions describe how to install PEDL for the first time; for directions on how to upgrade an existing PEDL installation, see the Upgrades section below.

Ensure that you are using the most up-to-date PEDL AMIs. Keep the AMI IDs handy as we will need them later (e.g., ami-0f4677bfc3161edc8).

Master

To install the master, we will launch an instance from the PEDL master AMI.

Let's start by navigating to the EC2 Dashboard of the AWS Console. Click "Launch Instance" and follow the instructions below:

  1. Choose AMI: find the PEDL Master AMI in "My AMIs" and click "Select".

  2. Choose Instance Type: we recommend a t2.medium or more powerful.

  3. Configure Instance: choose the IAM role according to these requirements.

  4. Add Storage: click Add New Volume and add an EBS volume of at least 100GB. If you have a previous PEDL installation that you are upgrading, you want to use the attach the same EBS volume as the previous installation. This volume will be used to store all your experiment metadata and checkpoints.

  5. Configure Security Group: choose or create a security group according to these Network Requirements.

  6. Review and launch the instance.

  7. SSH into the PEDL master and edit the config at /usr/local/pedl/etc/master.yaml according to the guide on Cluster Configuration.

  8. Start the PEDL master by entering make -C /usr/local/pedl enable-master into the terminal.

Agent

There is no installation needed for the Agent. The PEDL master will dynamically launch PEDL agent instances based on the Cluster Configuration.

Upgrades

Upgrading an existing PEDL installation with Dynamic Agents on AWS requires the same steps as an installation without dynamic agents. See upgrades.