Shortcuts

Install Determined on AWS

This document describes how to deploy a Determined cluster on Amazon Web Services (AWS). We provide the determined-deploy package for easy creation and deployment of these resources. If you would rather create the cluster manually, see the Manual Deployment section below.

For more information about using Determined on AWS, see the Determined on AWS topic guide.

determined-deploy Python Package

The determined-deploy package uses AWS CloudFormation to automatically deploy and configure a Determined cluster. CloudFormation builds the necessary components for Determined into a single CloudFormation stack.

Requirements

  • Either AWS credentials or an IAM role with permissions to access AWS CloudFormation APIs. See the AWS Documentation for information on how to use AWS credentials.

  • An AWS EC2 Keypair.

You may also want to increase the EC2 instance limits on your account — the default instance limits are quite low, particularly for GPU instances. For example, by default an AWS account can only create 128 vCPUs worth of P-type instances in a given AWS region. The default configuration for det-deploy can result in launching up to 5 p2.8xlarge instances (which have 32 vCPUs each), which would exceed the default quota. AWS instance limits can be increased by submitting a request to the AWS Support Center.

Installation

determined-deploy can be installed using pip:

pip install determined-deploy

Deploying

The basic command to deploy a cluster is as follows:

det-deploy aws up --cluster-id CLUSTER_ID --keypair KEYPAIR_NAME

CLUSTER_ID is an arbitrary unique ID for the new cluster. We recommend choosing a cluster ID that is memorable and helps identify what the cluster is being used for. The cluster ID will be used as the AWS CloudFormation stack name.

KEYPAIR_NAME is the name of the AWS EC2 key pair to use when provisioning the cluster. If the AWS CLI is installed on your machine, you can get a list of the available keypair names by running aws ec2 describe-key-pairs.

The deployment process may take 5–10 minutes. When it completes, summary information about the newly deployed cluster will be printed, including the URL of the Determined master.

Deployment Types

determined-deploy supports multiple different deployment types to work with different security needs. The deployment type can be specified using the --deployment-type argument (e.g., det-deploy aws --deployment-type secure).

  • simple: The simple deployment provides an easy way to deploy a Determined cluster in AWS. This creates the master instance in the default subnet for the account.

  • vpc: The VPC deployment creates a separate VPC with public subnets; the Determined cluster is deployed into these subnets.

  • efs: The EFS deployment creates an EFS drive and a Determined cluster into a separate VPC. The EFS drive attaches to agent instances at /mnt/efs.

  • fsx: The FSX deployment creates a Lustre FSx drive and a Determined cluster into a separate VPC. The FSx drive attaches to agent instances at /mnt/fsx.

  • secure: The secure deployment creates resources to lock down the Determined cluster. These resources are:

    • A VPC with a public and private subnet

    • A NAT gateway for the private subnet to make outbound connections

    • An S3 VPC gateway so the private subnet can access S3

    • A bastion instance in the public subnet

    • A master instance in the private subnet

CLI Arguments

Spinning up the Cluster

det-deploy aws up --cluster-id CLUSTER_ID --keypair KEYPAIR_NAME

Argument

Description

Default Value

--cluster-id

Unique ID for the cluster; it will be used as the CloudFormation stack name.

required

--keypair

The name of the AWS EC2 key pair to use for both master and agent instances.

required

--master-instance-type

AWS instance type to use for the master.

m5.large

--agent-instance-type

AWS instance type to use for the agents. Must be one of the following instance types: g4dn.xlarge, g4dn.2xlarge, g4dn.4xlarge, g4dn.8xlarge, g4dn.16xlarge, g4dn.12xlarge, g4dn.metal, p2.xlarge, p2.8xlarge, p2.16xlarge, p3.2xlarge, p3.8xlarge, p3.16xlarge, or p3dn.24xlarge.

p2.8xlarge

--deployment-type

The deployment template to use.

simple

--inbound-cidr

CIDR range for inbound traffic.

0.0.0.0/0

--db-password

The password for postgres user for database.

postgres

--region

AWS region to deploy into.

The default region for the AWS user

--max-idle-agent-period

The length of time to wait before idle dynamic agents will be automatically terminated.

10m (10 minutes)

--max-dynamic-agents

Maximum number of dynamic agent instances at one time.

5

--spot

Use spot instances.

False

--spot-max-price

A maximum price for the spot instances. If the price on the spot market exceeds this value, Determined will not create new instances. Optional

The on-demand price for the instance type.

--dry-run

Print the template but do not execute it.

False

Tearing Down the Cluster

det-deploy aws down --cluster-id CLUSTER_ID

Argument

Description

Default Value

--cluster-id

Unique ID for the cluster; it will be used for the CloudFormation stack name.

required

Manual Deployment

Database

Determined requires a PostgreSQL-compatible database, such as AWS Aurora. Configure the cluster to use the database by including the database information in master.yaml. Make sure to create a database before running the Determined cluster (e.g., CREATE DATABASE <database-name>).

Example master.yaml snippet:

db:
  user: "${database-user}"
  password: "${database-password}"
  host: "${database-hostname}"
  port: 5432
  name: "${database-name}"

Security Groups

VPC Security Groups provide a set of rules for inbound and outbound network traffic. The requirements for a Determined cluster are:

Master

  • Egress on all ports to agent security group

  • Egress outbound to the Internet

  • Ingress on port 8080 to view to the Determined WebUI

  • Ingress on port 22 for SSH (not required but strongly advised)

  • Ingress on all ports from agent security group

Example:

MasterSecurityGroupEgress:
  Type: AWS::EC2::SecurityGroupEgress
  Properties:
    GroupId: !GetAtt MasterSecurityGroup.GroupId
    DestinationSecurityGroupId: !GetAtt AgentSecurityGroup.GroupId
    FromPort: 0
    ToPort: 65535
    IpProtocol: tcp

MasterSecurityGroupInternet:
  Type: AWS::EC2::SecurityGroupEgress
  Properties:
    GroupId: !GetAtt MasterSecurityGroup.GroupId
    CidrIp: 0.0.0.0/0
    FromPort: 0
    ToPort: 65535
    IpProtocol: tcp

MasterSecurityGroupIngress:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !GetAtt MasterSecurityGroup.GroupId
    FromPort: 8080
    ToPort: 8080
    IpProtocol: tcp
    SourceSecurityGroupId: !GetAtt AgentSecurityGroup.GroupId

MasterSecurityGroupIngressUI:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !GetAtt MasterSecurityGroup.GroupId
    FromPort: 8080
    ToPort: 8080
    IpProtocol: tcp
    CidrIp: !Ref InboundCIDRRange

MasterSSHIngress:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !GetAtt MasterSecurityGroup.GroupId
    IpProtocol: tcp
    FromPort: 22
    ToPort: 22
    CidrIp: !Ref InboundCIDRRange

Agent

  • Egress on all ports to the Internet

  • Ingress on all ports from master security group

  • Ingress on all ports from agent security group

  • Ingress on port 22 for SSH (not required but strongly advised)

Example:

AgentSecurityGroupEgress:
  Type: AWS::EC2::SecurityGroupEgress
  Properties:
    GroupId: !GetAtt AgentSecurityGroup.GroupId
    CidrIp: 0.0.0.0/0
    FromPort: 0
    ToPort: 65535
    IpProtocol: tcp

AgentSecurityGroupIngressMaster:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !GetAtt AgentSecurityGroup.GroupId
    FromPort: 0
    ToPort: 65535
    IpProtocol: tcp
    SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId

AgentSecurityGroupIngressAgent:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !GetAtt AgentSecurityGroup.GroupId
    FromPort: 0
    ToPort: 65535
    IpProtocol: tcp
    SourceSecurityGroupId: !GetAtt AgentSecurityGroup.GroupId

AgentSSHIngress:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !GetAtt AgentSecurityGroup.GroupId
    IpProtocol: tcp
    FromPort: 22
    ToPort: 22
    CidrIp: !Ref InboundCIDRRange

IAM Roles

IAM roles comprise IAM policies, which provide access to AWS APIs such as the EC2 or S3 API. The IAM policies needed for the Determined cluster are:

Master

  • Allow EC2 to assume role

  • Allow EC2 to describe, create, and terminate instances with agent role

  • Allow EC2 to describe, create, and terminate spot instance requests (only required if using spot instances)

MasterRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - ec2.amazonaws.com
          Action:
            - sts:AssumeRole
    Policies:
      - PolicyName: determined-agent-policy
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action:
                - ec2:DescribeInstances
                - ec2:TerminateInstances
                - ec2:CreateTags
                - ec2:RunInstances
                - ec2:CancelSpotInstanceRequests      # Only required if using spot instances
                - ec2:RequestSpotInstances            # Only required if using spot instances
                - ec2:DescribeSpotInstanceRequests    # Only required if using spot instances
              Resource: "*"
      - PolicyName: pass-role
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action: iam:PassRole
              Resource: !GetAtt AgentRole.Arn

Agent

  • Allow EC2 to assume role

  • Allow S3 access for checkpoint storage

  • Allow agent instance to describe instances

AgentRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - ec2.amazonaws.com
          Action:
            - sts:AssumeRole
    Policies:
      - PolicyName: agent-s3-policy
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action: "s3:*"
              Resource: "*"
      - PolicyName: determined-ec2
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action:
                - ec2:DescribeInstances
              Resource: "*"

Master Node

The master node should be deployed on an EC2 instance with at least 4 CPUs (Intel Broadwell or later), 8GB of RAM, and 200GB of disk storage. This roughly corresponds to an EC2 t2.large instance or better. The AMI should be the default Ubuntu 18.04 AMI.

Running Determined

  1. Install Docker and create the determined Docker network.

    apt-get remove docker docker-engine docker.io containerd runc
    apt-get update
    apt-get install -y \
      apt-transport-https \
      ca-certificates \
      curl \
      gnupg-agent \
      software-properties-common
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
    add-apt-repository \
      "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
      $(lsb_release -cs) \
      stable"
    apt-get update
    apt-get install -y docker-ce docker-ce-cli containerd.io
    
    docker network create determined
    
  2. Configure the cluster with master.yaml. See Cluster Configuration for more information.

    Notes:

    • image_id should be the latest Determined agent AMI.

    • instance_type should be any p2 or p3 EC2 instance type.

    Warning

    An important assumption of Determined with Dynamic Agents is that any EC2 instances with the configured tag_key:tag_value pair are managed by the Determined master. This pair should be unique to your Determined installation. If it is not, Determined may inadvertently manage your non-Determined EC2 instances.

    If using spot instances, Determined also assumes that any EC2 spot instance requests with the configured tag_key:tag_value pair are managed by the Determined master.

    checkpoint_storage:
      type: s3
      bucket: ${CheckpointBucket}
    
    db:
      user: postgres
      password: "${DBPassword}"
      host: "${Database.Endpoint.Address}"
      port: 5432
      name: determined
    
    provisioner:
      iam_instance_profile_arn: ${AgentInstanceProfile.Arn}
      image_id: ${AgentAmiId}
      agent_docker_image: determinedai/determined-agent:${Version}
      instance_name: determined-agent-${UserName}
      instance_type: ${AgentInstanceType}
      master_url: http://local-ipv4:8080
      max_idle_agent_period: ${MaxIdleAgentPeriod}
      max_instances: ${MaxInstances}
      network_interface:
        public_ip: true
        security_group_id: ${AgentSecurityGroup.GroupId}
      provider: aws
      ssh_key_name: ${Keypair}
      tag_key: determined-${UserName}
      tag_value: determined-${UserName}-agent
    
  3. Start the Determined master.

    docker run \
      --rm \
      --network determined \
      -p 8080:8080 \
      -v master.yaml:/etc/determined/master.yaml \
      determinedai/determined-master:${Version}
    

Once you have your Determined cluster running on AWS, try out some of our tutorials.