Network Requirements

This document describes the networking requirements for PEDL clusters.

Performance

For best performance, we recommend that the PEDL master and agents use the same physical network or VPC.

Internet Access

  • The PEDL Docker images are hosted on Docker Hub. PEDL agents need access to Docker Hub, e.g., to build new images for user workloads.

  • If packages, data, or other resources needed by user workloads are hosted on the public Internet, PEDL agents will need to be able to access them. Note that agents can be configured to use proxies when accessing network resources.

Note

When using VPCs on a public cloud provider, additional steps may need to be taken to ensure that instances in the VPC can access the Internet:

  • On GCP, the instances need to have an external IP address or a GCP Cloud NAT should be configured for the VPC.
  • On AWS, the instances need to have a public IP address and a VPC Internet Gateway should be configured for the VPC.

Firewall Rules

The firewall rules must satisfy the following requirements.

Master

The PEDL master needs the following network access:

  • Inbound TCP to the master's HTTP and/or HTTPS ports from the PEDL agent instances, as well as all machines where developers want to use the PEDL CLI. The default HTTP port is 8080. If HTTPS is enabled, the default HTTPS port is 8443.
  • Outbound TCP to all ports on the PEDL agents.

Agents

PEDL agents need the following network access:

  • Inbound TCP from all ports on the master to all ports on the agent.
  • Outbound TCP from all ports on the agent to the master's HTTP port (8080 by default).
  • Inbound and outbound TCP on all ports to and from each PEDL agent.
  • Outbound TCP to the services that host the Docker images, packages, data, and other resources that need to be accessed by user workloads.
    • For example, if your data is stored on Amazon S3, ensure the firewall rules allow access to this data.