Transformers#
The Huggingface transformers library is the de facto library for natural language processing (NLP) models. It provides pretrained weights for leading NLP models and lets you easily use these pretrained models for the most common NLP tasks, such as language modeling, text classification, and question answering.
model-hub makes it easy to train transformer models in Determined while keeping the developer experience as close as possible to working directly with transformers. The Determined library serves as an alternative to the HuggingFace Trainer Class and provides access to the benefits of using Determined, including:
Easy multi-node distributed training with no code modifications. Determined automatically sets up the distributed backend for you.
Experiment monitoring and tracking, artifact tracking, and state-of-the-art hyperparameter search without requiring third-party integrations.
Automated cluster management, fault tolerance, and job rescheduling to free you from provisioning resources closely monitoring experiments.
Note
To learn more about distributed training with Determined, visit the conceptual overview or the intro to implementing distributed training.
Model Hub Transformers is similar to the no_trainer version of transformers examples in that
you have more control over the training and evaluation routines if you want.
Given the above benefits, this library can be particularly useful if any of the following apply:
You are an Determined user that wants to get started quickly with transformers.
You are a transformers user that wants to easily run more advanced workflows like multi-node distributed training and advanced hyperparameter search.
You are a transformers user looking for a single platform to manage experiments, handle checkpoints with automated fault tolerance, and perform hyperparameter search/visualization.
Getting Started#
The easiest way to use Model Hub Transformers is to start with an existing example Trial. Model Hub Transformers includes thoroughly tested implementations of all core transformers tasks.
Model Hub Transformers Trials are infinitely customizable. See the Model Hub Transformers Tutorial to learn how to customize or build a Trial.
Limitations#
The following HuggingFace transformers features are currently not supported:
TensorFlow version of transformers
Support for fairscale
Running on TPUs