Hyperparameter Search: Adaptive (Asynchronous)

The adaptive_asha search method employs the same underlying algorithm as the Adaptive (Advanced) method, but it uses an asynchronous version of successive halving (ASHA), which is more suitable for large-scale experiments with hundreds or thousands of trials.

Quick start

Here are some suggested initial settings for adaptive_asha that typically work well.

Search mode:

• mode: Set to standard.

Resource budget:

• target_trial_steps: The maximum number of steps that any trial that survives to the end of the experiment will be trained for (a step is a fixed number of batches). This quantity is domain-specific and should roughly reflect the number of training steps needed for the model to converge on the data set. For users who would like to determine this number experimentally, train a model with reasonable hyperparameters using the single search method.

• max_trials: This indicates the total number of hyperparameter settings that will be evaluated in the experiment. Set max_trials to at least 500 to take advantage of speedups from early-stopping. You can also set a large max_trials and stop the experiment once the desired performance is achieved.

• max_concurrent_trials: This field controls the degree of parallelism of the experiment. The experiment will have a maximum of this many trials training simultaneously at any one time. The adaptive_asha searcher scales nearly perfectly with additional compute, so you should set this field based on compute environment constraints.