PEDL consists of several components:

  • a master that schedules workloads and stores metadata

  • one or more agents that run workloads, typically using GPUs

  • a command-line tool (called pedl) that ML developers can use to launch new workloads and interact with PEDL.

The PEDL master and agents should typically be installed and configured by a system administrator; see the administrator install guide for more information.

Each ML developer that wants to use PEDL should install a copy of the PEDL CLI on their local development machine. The CLI is distributed as a Python wheel. You can install this wheel as follows:

pip install pedl-*.whl

We suggest installing the CLI into a virtualenv, although this is optional. To install the CLI into a virtualenv, first activate the virtualenv and then type the command above.

After the CLI has been installed, it should be configured to connect to the PEDL master at the appropriate IP address. This can be accomplished by setting the PEDL_MASTER environmental variable:

export PEDL_MASTER=<master IP>

You may want to place this into the appropriate configuration file for your login shell (e.g., .bashrc).

More information about using the PEDL CLI can be found with the command pedl --help.