DeepSpeed API

DeepSpeed is a Microsoft library that supports large-scale, distributed learning with sharded optimizer state training and pipeline parallelism. Determined supports DeepSpeed with the DeepSpeedTrial API. DeepSpeedTrial provides a way to use an automated training loop with DeepSpeed.

Determined DeepSpeed documentation:

• Usage Guide guides you through how to subclass DeepSpeedTrial for your own training experiments.

• Advanced Usage discusses advanced topics like using multiple model engines, manual gradient aggregation, custom data loaders, and custom model parallelism.

• PyTorchTrial to DeepSpeedTrial covers how to convert an existing PyTorchTrial to DeepSpeedTrial.

• API Reference lays out the classes and methods related to DeepSpeed support including the full API specification for DeepSpeedTrial and DeepSpeedTrialContext.