Install Determined on AWS¶
This document describes how to deploy a Determined cluster on Amazon Web Services (AWS). We provide the determined-deploy package for easy creation and deployment of these resources. If you would rather create the cluster manually, see the Manual Deployment section below.
For more information about using Determined on AWS, see the Determined on AWS topic guide.
determined-deploy
Python Package¶
The determined-deploy
package uses AWS CloudFormation to automatically deploy
and configure a Determined cluster. CloudFormation builds the necessary components
for Determined into a single CloudFormation stack.
Requirements¶
Either AWS credentials or an IAM role with permissions to access AWS CloudFormation APIs. See the AWS Documentation for information on how to use AWS credentials.
An AWS EC2 Keypair.
Deploying¶
The basic command to deploy a cluster is as follows:
det-deploy aws up --cluster-id CLUSTER_ID --keypair KEYPAIR
CLUSTER_ID
is an arbitrary unique ID for the new cluster.
We recommend choosing a cluster ID that is memorable and
helps identify what the cluster is being used for. The cluster ID will be
used as the AWS CloudFormation stack name.
KEYPAIR
is the name of the AWS EC2 key pair to use when provisioning
the cluster. If the AWS CLI is installed on your machine, you can get a
list of the available keypair names by running aws ec2 describe-key-pairs
.
The deployment process may take 5–10 minutes. When it completes, summary information about the newly deployed cluster will be printed, including the URL of the Determined master.
Deployment Types¶
determined-deploy
supports multiple different deployment types to
work with different security needs. The deployment type can be specified
using the --deployment-type
argument (e.g., det-deploy
aws --deployment-type secure
).
simple
: The simple deployment provides an easy way to deploy a Determined cluster in AWS. This creates the master instance in the default subnet for the account.vpc
: The VPC deployment creates a separate VPC with public subnets; the Determined cluster is deployed into these subnets.secure
: The secure deployment creates resources to lock down the Determined cluster. These resources are:A VPC with a public and private subnet
A NAT gateway for the private subnet to make outbound connections
An S3 VPC gateway so the private subnet can access S3
A bastion instance in the public subnet
A master instance in the private subnet
CLI Arguments¶
Spinning up the Cluster¶
det-deploy aws up --cluster-id CLUSTER_ID --keypair KEYPAIR
Argument |
Description |
Default Value |
---|---|---|
|
Unique ID for the cluster; it will be used as the CloudFormation stack name. |
required |
|
The name of the AWS EC2 key pair to use for both master and agent instances. |
required |
|
Instance type for master instance. |
m5.large |
|
Instance type for agent instances. |
p2.8xlarge |
|
The deployment template to use. |
simple |
|
CIDR range for inbound traffic. |
0.0.0.0/0 |
|
The password for |
postgres |
|
AWS region to deploy into. |
The default region for the AWS user |
|
The length of time to wait before idle dynamic agents will be automatically terminated. |
10m (10 minutes) |
|
Maximum number of dynamic agent instances at one time. |
5 |
|
Print the template but do not execute it. |
False |
Tearing Down the Cluster¶
det-deploy aws down --cluster-id CLUSTER_ID
Argument |
Description |
Default Value |
---|---|---|
|
Unique ID for the cluster; it will be used for the CloudFormation stack name. |
required |
Manual Deployment¶
Database¶
Determined requires a PostgreSQL-compatible database, such as AWS Aurora. Configure the cluster to use the database by
including the database information in master.yaml
. Make sure to create a database before running
the Determined cluster (e.g., CREATE DATABASE <database-name>
).
Example master.yaml
snippet:
db:
user: "${database-user}"
password: "${database-password}"
host: "${database-hostname}"
port: 5432
name: "${database-name}"
Security Groups¶
VPC Security Groups provide a set of rules for inbound and outbound network traffic. The requirements for a Determined cluster are:
Master¶
Egress on all ports to agent security group
Egress outbound to the Internet
Ingress on port 8080 to view to the Determined WebUI
Ingress on port 22 for SSH (not required but strongly advised)
Ingress on all ports from agent security group
Example:
MasterSecurityGroupEgress:
Type: AWS::EC2::SecurityGroupEgress
Properties:
GroupId: !GetAtt MasterSecurityGroup.GroupId
DestinationSecurityGroupId: !GetAtt AgentSecurityGroup.GroupId
FromPort: 0
ToPort: 65535
IpProtocol: tcp
MasterSecurityGroupInternet:
Type: AWS::EC2::SecurityGroupEgress
Properties:
GroupId: !GetAtt MasterSecurityGroup.GroupId
CidrIp: 0.0.0.0/0
FromPort: 0
ToPort: 65535
IpProtocol: tcp
MasterSecurityGroupIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt MasterSecurityGroup.GroupId
FromPort: 8080
ToPort: 8080
IpProtocol: tcp
SourceSecurityGroupId: !GetAtt AgentSecurityGroup.GroupId
MasterSecurityGroupIngressUI:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt MasterSecurityGroup.GroupId
FromPort: 8080
ToPort: 8080
IpProtocol: tcp
CidrIp: !Ref InboundCIDRRange
MasterSSHIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt MasterSecurityGroup.GroupId
IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: !Ref InboundCIDRRange
Agent¶
Egress on all ports to the Internet
Ingress on all ports from master security group
Ingress on all ports from agent security group
Ingress on port 22 for SSH (not required but strongly advised)
Example:
AgentSecurityGroupEgress:
Type: AWS::EC2::SecurityGroupEgress
Properties:
GroupId: !GetAtt AgentSecurityGroup.GroupId
CidrIp: 0.0.0.0/0
FromPort: 0
ToPort: 65535
IpProtocol: tcp
AgentSecurityGroupIngressMaster:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt AgentSecurityGroup.GroupId
FromPort: 0
ToPort: 65535
IpProtocol: tcp
SourceSecurityGroupId: !GetAtt MasterSecurityGroup.GroupId
AgentSecurityGroupIngressAgent:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt AgentSecurityGroup.GroupId
FromPort: 0
ToPort: 65535
IpProtocol: tcp
SourceSecurityGroupId: !GetAtt AgentSecurityGroup.GroupId
AgentSSHIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: !GetAtt AgentSecurityGroup.GroupId
IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: !Ref InboundCIDRRange
IAM Roles¶
IAM roles comprise IAM policies, which provide access to AWS APIs such as the EC2 or S3 API. The IAM policies needed for the Determined cluster are:
Master¶
Allow EC2 to assume role
Allow EC2 to dynamically create and terminate instances with agent role
MasterRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: determined-agent-policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- ec2:DescribeInstances
- ec2:TerminateInstances
- ec2:CreateTags
- ec2:RunInstances
Resource: "*"
- PolicyName: pass-role
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: iam:PassRole
Resource: !GetAtt AgentRole.Arn
Agent¶
Allow EC2 to assume role
Allow S3 access for checkpoint storage
Allow agent instance to describe instances
AgentRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: agent-s3-policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: "s3:*"
Resource: "*"
- PolicyName: determined-ec2
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- ec2:DescribeInstances
Resource: "*"
Master Node¶
The master node should be deployed on an EC2 instance with at least 4 CPUs (Intel Broadwell or later), 8GB of RAM, and 200GB of disk storage. This roughly corresponds to an EC2 t2.large instance or better. The AMI should be the default Ubuntu 18.04 AMI.
Running Determined¶
Install Docker and create the
determined
Docker network.apt-get remove docker docker-engine docker.io containerd runc apt-get update apt-get install -y \ apt-transport-https \ ca-certificates \ curl \ gnupg-agent \ software-properties-common curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" apt-get update apt-get install -y docker-ce docker-ce-cli containerd.io docker network create determined
Configure the cluster with
master.yaml
. See Cluster Configuration for more information.Notes:
agent-ami
should be the latest Determined agent AMI.instance-type
should be any p2 or p3 EC2 instance type.
Warning
An important assumption of Determined with Dynamic Agents is that any EC2 instances with the configured tag_key:tag_value pair are managed by the Determined master. This pair should be unique to your Determined installation. If it is not, Determined may inadvertently manage your non-Determined EC2 instances.
checkpoint_storage: type: s3 bucket: ${CheckpointBucket} save_experiment_best: 0 save_trial_best: 1 save_trial_latest: 1 db: user: postgres password: "${DBPassword}" host: "${Database.Endpoint.Address}" port: 5432 name: determined provisioner: iam_instance_profile_arn: ${AgentInstanceProfile.Arn} image_id: ${AgentAmiId} agent_docker_image: determinedai/determined-agent:${Version} instance_name: determined-agent-${UserName} instance_type: ${AgentInstanceType} master_url: http://local-ipv4:8080 max_idle_agent_period: ${MaxIdleAgentPeriod} max_instances: ${MaxInstances} network_interface: public_ip: true security_group_id: ${AgentSecurityGroup.GroupId} provider: aws root_volume_size: 200 ssh_key_name: ${Keypair} tag_key: determined-${UserName} tag_value: determined-${UserName}-agent
Start the Determined master.
docker run \ --rm \ --network determined \ -p 8080:8080 \ -v master.yaml:/etc/determined/master.yaml \ determinedai/determined-master:${Version}
Once you have your Determined cluster running on AWS, try out some of our tutorials.