DeepSpeed is a Microsoft library that supports large-scale,
distributed learning with sharded optimizer state training and pipeline parallelism. Determined
supports DeepSpeed with the
DeepSpeedTrial provides a way to use an automated
training loop with DeepSpeed.
Determined DeepSpeed documentation:
Advanced Usage discusses advanced topics like using multiple model engines, manual gradient aggregation, custom data loaders, and custom model parallelism.