Notifications#

Monitoring experiment status is crucial when working with Determined. To integrate Determined into your existing workflows, you can use Workload Alerting through webhooks. This feature allows you to receive timely updates about your experiments via various channels such as email, Slack messages, or other systems.

Workload alerting is particularly useful for real-time monitoring, debugging, and custom notifications. For example, you can configure alerts to trigger as soon as specific events occur in your experiments, rather than waiting for tasks to reach final states like “Completed” or “Error”.

Webhooks such as tasklog webhooks are useful for real-time monitoring, debugging, custom notifications, and integration with other systems. For example, using Tasklog, you could get updates as they happen rather than wait for the task to reach a final state such as Completed or Error.

Security and Signed Payload#

Each webhook request will include a signed payload that users can use to verify that webhook calls from Determined are genuine. This will require that users define a webhook_signing_key for signing. If a webhook signing key is not provided then one will be generated for the user.

Setting the Key#

The signing key can be set in the following ways:

In the master yaml configuration file by adding:

webhooks:
   signing_key: <signing_key>

By setting a DET_WEBHOOKS_SIGNING_KEY environment variable.
By specifying a ---security-webhooks-signing-key flag.

Retrieving the Key#

The key can be found in the cluster configuration. For example it will be returned in api/v1/master/config.

Event Payload#

Determined supports two types of webhooks: Slack and Default. A payload for a Default webhook will contain information about the event itself, the trigger for the event, and the entity that triggered the event. The shape of event_data is determined by event_type. Below is an example payload for EXPERIMENT_STATE_CHANGE; other types may be structured differently.

{
  "event_id": "4cd26e62-60c6-4a8b-8d03-7629091a4ef5",   // unique event UUID
  "event_type": "EXPERIMENT_STATE_CHANGE", // the trigger type for the event
  "timestamp": 1665689991, // the timestamp of the event occurrence
  "condition": {
    "state": "COMPLETED" // the condition that triggered the event; for now the condition will always be of this format and contain either "COMPLETED" or "ERRORED"
  },
  "event_data": {
    "experiment": {
      "id": 41,
      "state": "COMPLETED",
      "name": "cifar10_pytorch_const",
      "slots_per_trial": 2,
      "duration": 41, // experiment duration measured in seconds
      "resource_pool": "default",
      "workspace": "test workspace", // experiment workspace name
      "project": "test project" // experiment project name
   }
  }
}

Signed Payload#

For every webhook request Determined will generate two headers, X-Determined-AI-Signature and X-Determined-AI-Signature-Timestamp, which can be used to verify each request to a webhook endpoint.

The X-Determined-AI-Signature-Timestamp will represent the time at which the request was generated and sent. You can choose to inspect this timestamp and decide whether to discard any requests with a timestamp that is too distant from the current time.
The X-Determined-AI-Signature will be a representation of a “signed” request payload. The signed request payload will be generated in the following way:
- Combine the timestamp in X-Determined-AI-Signature-Timestamp, the comma character “,” and the request body, which will be the entire event payload.
- Create an HMAC using SHA256 hashing, with the webhook_signing_key and the event payload from previous step.
You can then check to make sure the X-Determined-AI-Signature header value and the generated signed payload match.

Below is an example of handling a signed payload in Python.

import hashlib, hmac, json

# User-defined function to authenticate webhook requests
def authenticate_webhook_request(request_body, request_headers, webhook_signing_key):
    timestamp = request_headers["X-Determined-AI-Signature-Timestamp"]
    signed_payload = request_headers["X-Determined-AI-Signature"]
    request_body = json.dumps(separators=(",", ":"), obj=json.loads(request_body))
    calculated_signed_payload = hmac.new(
        webhook_signing_key.encode(),
        f"{timestamp},{request_body}".encode(),
        digestmod=hashlib.sha256,
    ).hexdigest()
    return calculated_signed_payload == signed_payload

Note

The request body in the function shown above will be the JSON payload from the request. Ensure that the JSON payload does not contain spaces between keys and their values when creating the signed payload. For example “{“key_one”: “value_one”}” will fail authentication, while “{“key_one”:”value_one”}” will yield the correct signed payload value.

Supported Triggers#

Determined supports the following webhook trigger types:

COMPLETED or ERROR will be triggered when an experiment in scope is completed or errored.

TASKLOG will be triggered when a task matching regex is detected.

CUSTOM will only be triggered from experiment code.

# Example code to trigger a custom trigger.

# config.yaml
integrations:
   webhooks:
      webhook_name:
         - <webhook_name>

# code.py
with det.core.init() as core_context:
   core_context.alert(title="some title", description="some description", level="info")

Using Webhooks#

To get started with webhooks in Determined:

For step-by-step instructions on creating webhooks, see Creating Webhooks.
For use cases and best practices, visit Workload Alerting guide.
For platform-specific integration guides, see:
- Slack
- Zapier