The master allocates cluster resources (slots) among the active experiments using a fair-share scheduling policy. In other words, slots are divided among the active experiments according to the demand (number of desired concurrent tasks) of each experiment. For instance, in an eight GPU cluster with two experiments with demands of three and six respectively, the scheduler assigns three slots and five slots respectively. As new experiments become active or the resource demand of an active experiment changes, the scheduler will adjust how slots are allocated to experiments as apropriate.
Scheduling behavior can be configured via the
resources section of the experiment config file; see the configuration documentation for details.