Using The Determined REST APIs

Determined’s REST APIs provide a way for users and external tools to interact with a Determined cluster programmatically. Determined includes detailed documentation about all of the REST endpoints provided by the API, alongside a playground for interacting with the API. The endpoint documentation and playground are provided via Swagger UI.

Swagger UI

As Swagger puts it:

“Swagger UI allows anyone — be it your development team or your end consumers — to visualize and interact with the API’s resources without having any of the implementation logic in place. It’s automatically generated from your OpenAPI (formerly known as Swagger) Specification, with the visual documentation making it easy for back end implementation and client-side consumption.”

A static version of the Swagger UI is bundled with these docs here. If you have access to a running Determined cluster you can checkout the live-interact version by clicking the API icon from the Determined WebUI or by visiting the path /docs/rest-api/ on your Determined cluster.

The Swagger UI gives you an up-to-date view of the APIs your Determined cluster provides in a readable and interactive fashion.

On this page, you’ll see a list of all the available endpoints grouped by their workflows. If you click on each endpoint you’ll be presented with expected inputs and outputs for it as well as an interactive tool that lets you try out the endpoint. Note that for the interactive portion to work, you’d need to have the appropriate cookie set and have a running cluster. To set up the authentication take a look at the authentication section below or simply log in through the WebUI.

Authentication

Most of the API calls to Determined cluster need to be authenticated. On each API call, the server expects a Bearer token to be present.

To receive a token POST a valid username and password combination to the login endpoint, currently at /api/v1/auth/login with the following format:

{
  "username": "string",
  "password": "string"
}

Sample request:

curl -s "${DET_MASTER}/api/v1/auth/login" \
  -H 'Content-Type: application/json' \
  --data-binary '{"username":"determined","password":""}'

Sample response:

{
  "token": "string",
  "user": {
    "username": "string",
    "admin": true,
    "active": true,
    "agent_user_group": {
      "agent_uid": 0,
      "agent_gid": 0
    }
  }
}

Once we have the token we can store and attach to our future API calls under Authorization header and with the following format: Bearer $TOKEN.

Example

In this example, we will show how to use the REST APIs to unarchive an experiment that was previously archived.

By looking at the archive endpoint entry from our API docs (Swagger UI), we see that all we need is an experiment ID.

To find our an experiment that was archived, we lookup the experiments endpoint on our to figure out which filtering options we have. We see that we have archived and limit to work with. Using a bearer token we authenticate our request. We then use the archived and limit query parameters to limit the result set to only show a single archived experiment.

curl -H "Authorization: Bearer ${token}" "${DET_MASTER}/api/v1/experiments?archived=true&limit=1"

Here’s what our example response looks like. We see it matches the expected response shape.

{
  "experiments": [
    {
      "id": 16,
      "description": "mnist_pytorch_const",
      "labels": [],
      "startTime": "2020-08-26T20:12:35.337160Z",
      "endTime": "2020-08-26T20:12:51.951720Z",
      "state": "STATE_COMPLETED",
      "archived": true,
      "numTrials": 1,
      "progress": 0,
      "username": "determined"
    }
  ],
  "pagination": {
    "offset": 0,
    "limit": 1,
    "startIndex": 0,
    "endIndex": 1,
    "total": 1
  }
}

Now that we have our desired experiment’s ID, we use it to target the experiment through the unarchive endpoint using a POST request as specified by the endpoint:

curl -H "Authorization: Bearer ${token}" -X POST "${DET_MASTER}/api/v1/experiments/16/unarchive"

How Our REST APIs work

We use protobuf to define language-agnostic message structures. We use these type definitions with gRPC-gateway to provide consistent REST endpoints to serve various needs.

Using these two tools we also auto-generate OpenAPI v2 spec (aka Swagger RESTful API Documentation Specification) which lets us inline documentation for each endpoint and response in our codebase.

Having this specification prepared we can serve it to different tools to generate code for different languages (eg swagger codegen) as well as provide web-based explorers into our APIs (e.g., Swagger UI).