Optimizing Distributed and Parallel Training¶
The Distributed and Parallel Training how-to guide describes how to configure distributed and parallel training. In this topic guide, we will describe how to optimize distributed and parallel training, focusing on the following topics:
setting batch size,
single-machine parallel training,
multi-machine distributed training, and
configuring advanced optimizations.
Determined uses synchronous data parallelism for distributed and parallel training.
Setting Global Batch Size¶
When doing distributed and parallel training, the global_batch_size
specified in the Experiment Configuration is partitioned across
slots_per_trial GPUs. The per-GPU batch size is set to:
global_batch_size / slots_per_trial. If slots_per_trial does not divide the global_batch_size evenly,
the batch size is rounded down. For convenience, the
per-GPU batch size can be accessed via the Trial API, using
context.get_per_slot_batch_size.
For improved performance, we recommend weak-scaling: increasing your
global_batch_size proportionally with slots_per_trial (e.g.,
change global_batch_size of 32 for slots_per_trial of 1 to
global_batch_size of 128 for slots_per_trial of 4).
Adjusting global_batch_size can affect your model convergence, which can affect your training and/or
testing accuracy. You may need to adjust model hyperparameters like the learning rate and/or
use a different optimizer when training with larger batch sizes.
Single-Machine Parallelism¶
When using multi-GPU training in Determined, we recommend first trying single-machine parallel training, then multiple machine distributed training. For single-machine parallel training, in the experiment configuration, add:
resources:
slots_per_trial: N
where N is the total number of GPUs on an agent machine. In this configuration, trials will train using all the resources on a single machine.
Distributed Multi-Machine Parallelism¶
Multi-machine parallelism offers the ability to further parallelize training across more GPUs. For distributed multi-machine training, in the experiment configuration, add:
resources:
slots_per_trial: M
where M is a multiple of the total number of GPUs on an agent machine. For example, if your cluster consists of 8-GPU agent machines, valid values for M would be 16, 24, 32, etc. In this configuration, trials will use all the resources of multiple machines to train a model.
Warning
For distributed multi-machine training, Determined automatically detects
a common network interface shared by the agent machines. If your cluster has
multiple common network interfaces, please specify the fastest one
in Cluster Configuration under task_container_defaults.dtrain_network_interface.
Configuring Advanced Optimizations¶
Determined offers several optimizations to further reduce
training time. These optimizations are available in Experiment Configuration
under optimizations.
The optimizations.aggregation_frequency option
evaluates more batches before exchanging gradients and is helpful
in situations where it is not possible to increase the batch size
directly. This optimization increases your effective batch size to
aggregation_frequency * global_batch_size.
The optimizations.gradient_compression option reduces
the time it takes to transfer gradients between GPUs.
The optimization.auto_tune_tensor_fusion option automatically identifies
the optimal message size during gradient transfers, reducing
communication overhead.
The average_training_metrics option will average the training metrics
across GPUs at the end of every step, which requires communication. Typically
this will not have an impact on training performance, but if you have a very
small batches_per_step, ensuring it is disabled may improve performance.
If you do not see improved performance, there might be a performance bottleneck in the model that cannot be directly alleviated by using multiple GPUs, e.g., data loading. We suggest experimenting with a synthetic dataset to verify performance of multi-GPU training.
Warning
Distributed and parallel training is designed to maximize performance by training with all the resources of a machine. This can lead to situations where an experiment is created but never becomes active: if the number of GPUs requested does not divide into the machines available, for instance, or if another experiment is already using some GPUs on a machine.
If a multi-GPU experiment does not become active after a minute or so,
please confirm that slots_per_trial is a multiple of the number of GPUs
available on a machine. Also, you can also use the CLI command det task
list to check if any other tasks are using GPUs and preventing your
experiment from using all the GPUs on a machine.