Jupyter Notebooks Tutorial

To launch a notebook, start by installing the PEDL command line interface on a development machine.

Once the CLI is installed, try launching your first notebook with the pedl notebook start command:

$ pedl notebook start
Scheduling notebook unique-oyster (id: 5b2a9ea4-a6bb-4d2b-b42b-25e4064a3220)...
[DOCKER BUILD 🔨] Step 1/11 : FROM nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04
[DOCKER BUILD 🔨]
[DOCKER BUILD 🔨]  ---> 9918ba890dca
[DOCKER BUILD 🔨] Step 2/11 : RUN rm /etc/apt/sources.list.d/*
...
[DOCKER BUILD 🔨] Successfully tagged nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04-73bf63cc864088137a477ce62f39ffe8
[PEDL] 2019-04-04T17:53:22.076591700Z [I 17:53:22.075 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
[PEDL] 2019-04-04T17:53:23.067911400Z [W 17:53:23.067 NotebookApp] All authentication is disabled.  Anyone who can connect to this server will be able to run code.
[PEDL] 2019-04-04T17:53:23.073644300Z [I 17:53:23.073 NotebookApp] Serving notebooks from local directory: /
disconnecting websocket
Jupyter Notebook is running at: http://localhost:8080/proxy/5b2a9ea4-a6bb-4d2b-b42b-25e4064a3220-notebook-0/lab/tree/Notebook.ipynb?reset

After the notebook has been scheduled onto the cluster, the PEDL CLI will open a web browser window pointed to that notebook's URL. Back in the terminal, you can use the pedl notebook list command to see that this notebook is one of those currently RUNNING on the PEDL cluster:

$ pedl notebook list
 Id                                   | Entry Point                                            | Registered Time              | State
--------------------------------------+--------------------------------------------------------+------------------------------+---------
 0f519413-2411-4b3c-adbc-9b1b60c96156 | ['jupyter', 'notebook', '--config', '/etc/jupyter.py'] | 2019-04-04T17:52:48.1961129Z | RUNNING
 5b2a9ea4-a6bb-4d2b-b42b-25e4064a3220 | ['jupyter', 'notebook', '--config', '/etc/jupyter.py'] | 2019-04-04T17:53:20.387903Z  | RUNNING
 66da599e-62d2-4c2d-91c4-01a04045e4ab | ['jupyter', 'notebook', '--config', '/etc/jupyter.py'] | 2019-04-04T17:52:58.4573214Z | RUNNING

Since the lifecycle management of Jupyter notebooks in PEDL is left up to the user, this notebook will continue running until it is explicitly shut down. To terminate the notebook, you can use the pedl notebook kill command:

$ pedl notebook kill 5b2a9ea4-a6bb-4d2b-b42b-25e4064a3220
Killed notebook 5b2a9ea4-a6bb-4d2b-b42b-25e4064a3220

Using the CLI in Notebooks

The PEDL CLI is installed into notebook containers by default. This allows users to interact with PEDL from inside a notebook -- e.g., to launch new deep learning workloads or examine the metrics from an active or historical PEDL experiment. For example, to list PEDL experiments from inside a notebook, run the notebook command !pedl experiment list.