Install Determined on AWS¶

This document describes how to deploy a Determined cluster on Amazon Web Services (AWS). We provide the determined-deploy package for easy creation and deployment of these resources. If you would rather create the cluster manually, see the Manual Deployment section below.

For more information about using Determined on AWS, see the Determined on AWS topic guide.

`determined-deploy` Python Package¶

The determined-deploy package uses AWS CloudFormation to automatically deploy and configure a Determined cluster. CloudFormation builds the necessary components for Determined into a single CloudFormation stack.

Requirements¶

Either AWS credentials or an IAM role with permissions to access AWS CloudFormation APIs. See the AWS Documentation for information on how to use AWS credentials.
An AWS EC2 Keypair.

You may also want to increase the EC2 instance limits on your account — the default instance limits are quite low, particularly for GPU instances. For example, by default an AWS account can only create 128 vCPUs worth of P-type instances in a given AWS region. The default configuration for det-deploy can result in launching up to 5 p2.8xlarge instances (which have 32 vCPUs each), which would exceed the default quota. AWS instance limits can be increased by submitting a request to the AWS Support Center.

Installation¶

determined-deploy can be installed using pip:

pip install determined-deploy

Deploying¶

The basic command to deploy a cluster is as follows:

det-deploy aws up --cluster-id CLUSTER_ID --keypair KEYPAIR_NAME

CLUSTER_ID is an arbitrary unique ID for the new cluster. We recommend choosing a cluster ID that is memorable and helps identify what the cluster is being used for. The cluster ID will be used as the AWS CloudFormation stack name.

KEYPAIR_NAME is the name of the AWS EC2 key pair to use when provisioning the cluster. If the AWS CLI is installed on your machine, you can get a list of the available keypair names by running aws ec2 describe-key-pairs.

The deployment process may take 5–10 minutes. When it completes, summary information about the newly deployed cluster will be printed, including the URL of the Determined master.

Deployment Types¶

determined-deploy supports multiple different deployment types to work with different security needs. The deployment type can be specified using the --deployment-type argument (e.g., det-deploy aws --deployment-type secure).

simple: The simple deployment provides an easy way to deploy a Determined cluster in AWS. This creates the master instance in the default subnet for the account.
vpc: The VPC deployment creates a separate VPC with public subnets; the Determined cluster is deployed into these subnets.
efs: The EFS deployment creates an EFS drive and a Determined cluster into a separate VPC. The EFS drive attaches to agent instances at /mnt/efs.
fsx: The FSX deployment creates a Lustre FSx drive and a Determined cluster into a separate VPC. The FSx drive attaches to agent instances at /mnt/fsx.
secure: The secure deployment creates resources to lock down the Determined cluster. These resources are:
- A VPC with a public and private subnet
- A NAT gateway for the private subnet to make outbound connections
- An S3 VPC gateway so the private subnet can access S3
- A bastion instance in the public subnet
- A master instance in the private subnet

CLI Arguments¶

Spinning up the Cluster¶

det-deploy aws up --cluster-id CLUSTER_ID --keypair KEYPAIR_NAME

Argument	Description	Default Value
`--cluster-id`	Unique ID for the cluster; it will be used as the CloudFormation stack name.	required
`--keypair`	The name of the AWS EC2 key pair to use for both master and agent instances.	required
`--master-instance-type`	AWS instance type to use for the master.	m5.large
`--agent-instance-type`	AWS instance type to use for the agents. Must be one of the following instance types: `g4dn.xlarge`, `g4dn.2xlarge`, `g4dn.4xlarge`, `g4dn.8xlarge`, `g4dn.16xlarge`, `g4dn.12xlarge`, `g4dn.metal`, `p2.xlarge`, `p2.8xlarge`, `p2.16xlarge`, `p3.2xlarge`, `p3.8xlarge`, `p3.16xlarge`, or `p3dn.24xlarge`.	p2.8xlarge
`--deployment-type`	The deployment template to use.	simple
`--inbound-cidr`	CIDR range for inbound traffic.	0.0.0.0/0
`--db-password`	The password for `postgres` user for database.	postgres
`--region`	AWS region to deploy into.	The default region for the AWS user
`--max-idle-agent-period`	The length of time to wait before idle dynamic agents will be automatically terminated.	10m (10 minutes)
`--max-dynamic-agents`	Maximum number of dynamic agent instances at one time.	5
`--spot`	Use spot instances.	False
`--spot-max-price`	A maximum price for the spot instances. If the price on the spot market exceeds this value, Determined will not create new instances. Optional	The on-demand price for the instance type.
`--dry-run`	Print the template but do not execute it.	False

Tearing Down the Cluster¶

det-deploy aws down --cluster-id CLUSTER_ID

Argument	Description	Default Value
`--cluster-id`	Unique ID for the cluster; it will be used for the CloudFormation stack name.	required

Manual Deployment¶

Database¶

Determined requires a PostgreSQL-compatible database, such as AWS Aurora. Configure the cluster to use the database by including the database information in master.yaml. Make sure to create a database before running the Determined cluster (e.g., CREATE DATABASE <database-name>).

Example master.yaml snippet:

db:
  user: "${database-user}"
  password: "${database-password}"
  host: "${database-hostname}"
  port: 5432
  name: "${database-name}"

Security Groups¶

VPC Security Groups provide a set of rules for inbound and outbound network traffic. The requirements for a Determined cluster are:

Master¶

Egress on all ports to agent security group
Egress outbound to the Internet
Ingress on port 8080 to view to the Determined WebUI
Ingress on port 22 for SSH (not required but strongly advised)
Ingress on all ports from agent security group

Example:

MasterSecurityGroupEgress:
  Type: AWS::EC2::SecurityGroupEgress
  Properties:
    GroupId: !GetAtt MasterSecurityGroup.GroupId
    DestinationSecurityGroupId: !GetAtt AgentSecurityGroup.GroupId
    FromPort: 0
    ToPort: 65535
    IpProtocol: tcp

MasterSecurityGroupInternet:
  Type: AWS::EC2::SecurityGroupEgress
  Properties:
    GroupId: !GetAtt MasterSecurityGroup.GroupId
    CidrIp: 0.0.0.0/0
    FromPort: 0
    ToPort: 65535
    IpProtocol: tcp

MasterSecurityGroupIngress:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !GetAtt MasterSecurityGroup.GroupId
    FromPort: 8080
    ToPort: 8080
    IpProtocol: tcp
    SourceSecurityGroupId: !GetAtt AgentSecurityGroup.GroupId

MasterSecurityGroupIngressUI:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !GetAtt MasterSecurityGroup.GroupId
    FromPort: 8080
    ToPort: 8080
    IpProtocol: tcp
    CidrIp: !Ref InboundCIDRRange

MasterSSHIngress:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !GetAtt MasterSecurityGroup.GroupId
    IpProtocol: tcp
    FromPort: 22
    ToPort: 22
    CidrIp: !Ref InboundCIDRRange

Agent¶

Egress on all ports to the Internet
Ingress on all ports from master security group
Ingress on all ports from agent security group
Ingress on port 22 for SSH (not required but strongly advised)

Example:

AgentSecurityGroupEgress:
  Type: AWS::EC2::SecurityGroupEgress
  Properties:
    GroupId: !GetAtt AgentSecurityGroup.GroupId
    CidrIp: 0.0.0.0/0
    FromPort: 0
    ToPort: 65535
    IpProtocol: tcp

AgentSecurityGroupIngressMaster:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !GetAtt AgentSecurityGroup.GroupId
    FromPort: 0
    ToPort: 65535
    IpProtocol: tcp
    SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId

AgentSecurityGroupIngressAgent:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !GetAtt AgentSecurityGroup.GroupId
    FromPort: 0
    ToPort: 65535
    IpProtocol: tcp
    SourceSecurityGroupId: !GetAtt AgentSecurityGroup.GroupId

AgentSSHIngress:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !GetAtt AgentSecurityGroup.GroupId
    IpProtocol: tcp
    FromPort: 22
    ToPort: 22
    CidrIp: !Ref InboundCIDRRange

IAM Roles¶

IAM roles comprise IAM policies, which provide access to AWS APIs such as the EC2 or S3 API. The IAM policies needed for the Determined cluster are:

Master¶

Allow EC2 to assume role
Allow EC2 to describe, create, and terminate instances with agent role
Allow EC2 to describe, create, and terminate spot instance requests (only required if using spot instances)

MasterRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - ec2.amazonaws.com
          Action:
            - sts:AssumeRole
    Policies:
      - PolicyName: determined-agent-policy
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action:
                - ec2:DescribeInstances
                - ec2:TerminateInstances
                - ec2:CreateTags
                - ec2:RunInstances
                - ec2:CancelSpotInstanceRequests      # Only required if using spot instances
                - ec2:RequestSpotInstances            # Only required if using spot instances
                - ec2:DescribeSpotInstanceRequests    # Only required if using spot instances
              Resource: "*"
      - PolicyName: pass-role
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action: iam:PassRole
              Resource: !GetAtt AgentRole.Arn

Agent¶

Allow EC2 to assume role
Allow S3 access for checkpoint storage
Allow agent instance to describe instances

AgentRole:
  Type: AWS::IAM::Role
  Properties:
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - ec2.amazonaws.com
          Action:
            - sts:AssumeRole
    Policies:
      - PolicyName: agent-s3-policy
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action: "s3:*"
              Resource: "*"
      - PolicyName: determined-ec2
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action:
                - ec2:DescribeInstances
              Resource: "*"

Master Node¶

The master node should be deployed on an EC2 instance with at least 4 CPUs (Intel Broadwell or later), 8GB of RAM, and 200GB of disk storage. This roughly corresponds to an EC2 t2.large instance or better. The AMI should be the default Ubuntu 18.04 AMI.

Running Determined¶

Install Docker and create the determined Docker network.

apt-get remove docker docker-engine docker.io containerd runc
apt-get update
apt-get install -y \
  apt-transport-https \
  ca-certificates \
  curl \
  gnupg-agent \
  software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io

docker network create determined

Configure the cluster with master.yaml. See Cluster Configuration for more information.

Notes:

image_id should be the latest Determined agent AMI.
instance_type should be any p2 or p3 EC2 instance type.

Warning

An important assumption of Determined with Dynamic Agents is that any EC2 instances with the configured tag_key:tag_value pair are managed by the Determined master. This pair should be unique to your Determined installation. If it is not, Determined may inadvertently manage your non-Determined EC2 instances.

If using spot instances, Determined also assumes that any EC2 spot instance requests with the configured tag_key:tag_value pair are managed by the Determined master.

checkpoint_storage:
  type: s3
  bucket: ${CheckpointBucket}

db:
  user: postgres
  password: "${DBPassword}"
  host: "${Database.Endpoint.Address}"
  port: 5432
  name: determined

provisioner:
  iam_instance_profile_arn: ${AgentInstanceProfile.Arn}
  image_id: ${AgentAmiId}
  agent_docker_image: determinedai/determined-agent:${Version}
  instance_name: determined-agent-${UserName}
  instance_type: ${AgentInstanceType}
  master_url: http://local-ipv4:8080
  max_idle_agent_period: ${MaxIdleAgentPeriod}
  max_instances: ${MaxInstances}
  network_interface:
    public_ip: true
    security_group_id: ${AgentSecurityGroup.GroupId}
  provider: aws
  ssh_key_name: ${Keypair}
  tag_key: determined-${UserName}
  tag_value: determined-${UserName}-agent

Start the Determined master.

docker run \
  --rm \
  --network determined \
  -p 8080:8080 \
  -v master.yaml:/etc/determined/master.yaml \
  determinedai/determined-master:${Version}

Once you have your Determined cluster running on AWS, try out some of our tutorials.

Install Determined on AWS¶

determined-deploy Python Package¶

Requirements¶

Installation¶

Deploying¶

Deployment Types¶

CLI Arguments¶

Spinning up the Cluster¶

Tearing Down the Cluster¶

Manual Deployment¶

Database¶

Security Groups¶

Master¶

Agent¶

IAM Roles¶

Master¶

Agent¶

Master Node¶

Running Determined¶

`determined-deploy` Python Package¶