Skip to content

Jupyter Notebooks

Warning

Jupyter Notebooks is currently an experimental feature.

PEDL offers the ability to easily launch Jupyter notebooks attached to one or more slots in the cluster. Jupyter notebooks offer a convenient interface to develop and/or debug model definition code, visualize results of trained model(s), or even manage the entire training lifecycle of a model manually.

Under the hood, PEDL will schedule a Jupyter notebook in a containerized environment on the cluster and proxy HTTP requests to and from the notebook container through the PEDL master. The lifecycle management of Jupyter notebooks in PEDL is left up to the user -- once a new Jupyter notebook has been scheduled onto the cluster, it will remain scheduled indefinitely until the user explicitly shuts down the notebook.

Quick Start

To launch a notebook, you'll first need to install the PEDL command line interface on a development machine.

Once the PEDL CLI is installed, try launching your first notebook with the pedl notebook start command:

$ pedl notebook start
Scheduling notebook unique-oyster (id: 5b2a9ea4-a6bb-4d2b-b42b-25e4064a3220)...
[DOCKER BUILD 🔨] Step 1/11 : FROM nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04
[DOCKER BUILD 🔨]
[DOCKER BUILD 🔨]  ---> 9918ba890dca
[DOCKER BUILD 🔨] Step 2/11 : RUN rm /etc/apt/sources.list.d/*
...
[DOCKER BUILD 🔨] Successfully tagged nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04-73bf63cc864088137a477ce62f39ffe8
[PEDL] 2019-04-04T17:53:22.076591700Z [I 17:53:22.075 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
[PEDL] 2019-04-04T17:53:23.067911400Z [W 17:53:23.067 NotebookApp] All authentication is disabled.  Anyone who can connect to this server will be able to run code.
[PEDL] 2019-04-04T17:53:23.073644300Z [I 17:53:23.073 NotebookApp] Serving notebooks from local directory: /
disconnecting websocket
Jupyter Notebook is running at: http://localhost:8080/proxy/5b2a9ea4-a6bb-4d2b-b42b-25e4064a3220-notebook-0/notebooks/Notebook.ipynb

After the notebook has been scheduled onto the cluster, the PEDL CLI will open a web browser window pointed to that notebooks URL. Back on the terminal, you can use the pedl notebook list command to see this notebook as part of those currently RUNNING on the PEDL cluster:

$ pedl notebook list
 Id                                   | Entry Point                                            | Registered Time              | State
--------------------------------------+--------------------------------------------------------+------------------------------+---------
 0f519413-2411-4b3c-adbc-9b1b60c96156 | ['jupyter', 'notebook', '--config', '/etc/jupyter.py'] | 2019-04-04T17:52:48.1961129Z | RUNNING
 5b2a9ea4-a6bb-4d2b-b42b-25e4064a3220 | ['jupyter', 'notebook', '--config', '/etc/jupyter.py'] | 2019-04-04T17:53:20.387903Z  | RUNNING
 66da599e-62d2-4c2d-91c4-01a04045e4ab | ['jupyter', 'notebook', '--config', '/etc/jupyter.py'] | 2019-04-04T17:52:58.4573214Z | RUNNING

Since the lifecycle management of Jupyter notebooks in PEDL is left up to the user, this notebook will remain scheduled on the slot it is explicitly shut down. To shut down the notebook and free up the slot, you can use the pedl notebook kill command:

$ pedl notebook kill 5b2a9ea4-a6bb-4d2b-b42b-25e4064a3220
Killed notebook 5b2a9ea4-a6bb-4d2b-b42b-25e4064a3220

One powerful PEDL feature is the ability to easily modify the desired dependencies of your notebook environment:

$ pedl notebook --config environment.tensorflow=1.10.0

More generally, notebooks may be supplied an optional notebook configuration] to configure the notebook's enviornment. In addition to the --config flag, configuration may also be supplied via a YAML file (--config-file):

$ cat > config.yaml << EOL
description: test-notebook
resources:
  slots: 2
environment:
  python: 3.6.8
  tensorflow: 1.10.0
  keras: 2.2.4
EOL
$ pedl notebook --config-file config.yaml

See the Notebook Configuration section for full documentation of the configuration schema. Note that notebooks share the same configuration schema as Commands.