Queue Management¶

The Determined Queue Management system extends scheduler functionality to offer better visibility and control over scheduling decisions. It does this using the Job Queue, which provides better information about job ordering, such as which jobs are queued, and permits dynamic job modification.

Queue Management is a new feature that is available to the fair share scheduler and the priority scheduler. Queue Management, described in detail in the following sections, shows all submitted jobs and their states, and lets you modify some configuration options, such as priority, position in the queue, and resource pool.

To begin managing job queues, navigate to the WebUI Job Queue section or use the det job set of CLI commands.

Job States¶

Queued jobs can be in the Queued or Scheduled state:

Queued: Job received but resources not allocated
Scheduled: Scheduled to run or running, and resources may have been allocated.

Completed or errored jobs are not counted as active and are omitted from this list.

Visualize the Job Queue¶

You can view the job queue using the CLI or WebUI. In the WebUI, click the Job Queue tab. In the CLI, use one of the following commands:

$ det job list
$ det job ls

These commands show the default resource pool queue. To view other resource pool queues, use the --resource-pool option, specifying the pool:

$ det job list --resource-pool compute-pool

For more information about the CLI options, see the CLI documentation or use the det job list -h command.

The WebUI and the CLI display a table of results, ordered by scheduling order. The scheduling order does not represent the job priority. In addition to job order, the table includes the job states and number of slots allocated to each job.

Modify the Job Queue¶

The job queue can be changed in the WebUI Job Queue section or by using the CLI det job update command. You can make changes on a per-job basis by selecting a job and a job operation. Available operations include:

changing priorities for resource pools using the priority scheduler
changing weights for resource pools using the fair share scheduler
changing the order of queued jobs
changing resource pools

WebUI¶

To modify the job queue in the Webui,

Go to the Job Queue section.
Find the job to modify.
Click the three dots in the right-most column of the job.
Find and click the Manage Job option.
Make the change you want on the pop-up page, and click OK.

CLI¶

To modify the job queue in the CLI, use the det job update command. Run det job update --help for more information. Example operations:

$ det job update jobID --priority 10
$ det job update jobID --resource-pool a100
$ det job update jobID --ahead-of jobID-2

To update a job in batch, provide updates as shown:

$ det job update-batch job1.priority=1 job2.resource-pool="compute" job3.ahead-of=job1

Example workflow:

$ det job list
   # | ID       | Type            | Job Name   | Priority | Submitted            | Slots (acquired/needed) | Status          | User
-----+--------------------------------------+-----------------+--------------------------+------------+---------------------------+---------
   0 | 0d714127 | TYPE_EXPERIMENT | first_job  |       42 | 2022-01-01 00:01:00  | 1/1                     | STATE_SCHEDULED | user1
   1 | 73853c5c | TYPE_EXPERIMENT | second_job |       42 | 2022-01-01 00:01:01  | 0/1                     | STATE_QUEUED    | user1

$ det job update 73853c5c --ahead-of 0d714127

$ det job list
   # | ID       | Type            | Job Name   | Priority | Submitted            | Slots (acquired/needed) | Status          | User
-----+--------------------------------------+-----------------+--------------------------+------------+---------------------------+---------
   0 | 73853c5c | TYPE_EXPERIMENT | second_job |       42 | 2022-01-01 00:01:01  | 1/1                     | STATE_SCHEDULED | user1
   1 | 0d714127 | TYPE_EXPERIMENT | first_job  |       42 | 2022-01-01 00:01:00  | 0/1                     | STATE_QUEUED    | user1

$ det job update-batch 73853c5c.priority=1 0d714127.priority=1

$ det job list
   # | ID       | Type            | Job Name   | Priority | Submitted            | Slots (acquired/needed) | Status          | User
-----+--------------------------------------+-----------------+--------------------------+------------+---------------------------+---------
   0 | 73853c5c | TYPE_EXPERIMENT | second_job |       1 | 2022-01-01 00:01:01  | 1/1                     | STATE_SCHEDULED | user1
   1 | 0d714127 | TYPE_EXPERIMENT | first_job  |       1 | 2022-01-01 00:01:00  | 0/1                     | STATE_QUEUED    | user1

Job Queue Constraints¶

There are a number of constraints associated with using the job queue to modify jobs:

The priority and fair share fields are mutually exclusive. The priority field is only active for the priority scheduler and the fair share field is only active for the fair share scheduler. It is not possible for both to be active simultaneously.
The ahead-of, behind-of, and WebUI Move to Top operations are only available for the priority scheduler and are not possible with the fair share scheduler. These operations are not yet fully supported for the Kubernetes priority scheduler.
The change resource pool operation can only be performed on experiments. To change the resource pool of other tasks, cancel the task and resubmit it.