Frequently Asked Questions

Installation

Which dependencies are required to install the PEDL command line interface?

You’ll need to have Python >= 3.5 and pip installed in your development environment. If you need to install a new version of Python, one tool you can use is pyenv.

How do I install the PEDL command line interface?

pip install https://releases.determined.ai/cli/pedl-{{ pedl.version }}-py35.py36.py37-none-any.whl

You’ll need to enter the username and password for your organization, which will be provided by Determined AI as part of the initial training session.

When trying to install the PEDL command line interface, I encounter this distutils error

Uninstalling a distutils installed project (...) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.

If a Python library has previously been installed in your environment with distutils or conda, pip may not be able to upgrade or downgrade the library to the version required by PEDL. There are two recommended solutions:

  1. Install PEDL command line interface into a fresh virtualenv with no previous Python packages installed.

  2. Use --ignore-installed with pip to force overwriting the library version(s).

pip install --ignore-installed https://releases.determined.ai/cli/pedl-<VERSION>-py35.py36.py37-none-any.whl

Multi-GPU Training

Why do my multi-GPU training experiments never start?

It might be that slots_per_trial in the experiment configuration is not a multiple of the number of GPUs on a machine or that there are running tasks preventing your multi-GPU trials from acquiring all the GPUs on a single machine. Consider adjusting slots_per_trial or terminating existing tasks to free up slots in your cluster.

See Multi-GPU Training for more details.

Why do my multi-machine training experiments appear to be stuck?

Multi-machine training requires that all machines be able to connect to each other directly. There may be firewall rules or network configuration that prevent machines in your cluster from communicating. Please check if agent machines can access each other outside of PEDL (e.g., use the ping or netcat tools).

More rarely, if agents have multiple network interfaces and some of them are not routable, PEDL may pick one of those interfaces rather than one that allows one agent to contact another. In this case, it is possible to set the network interface used for multi-GPU training explicitly in the Cluster Configuration.