Hyperparameter Search Constraints¶
Determined’s Hyperparameter (HP) Search Constraints API enables finer-grained control over the hyperparameter search space through the specification of additional constraints. This functionality is particularly useful for incorporating prior knowledge/domain expertise into a hyperparameter search and constraining the search to models that fit a particular deployment environment.
Using Determined’s HP Search Constraints requires no changes to the configuration files. Rather,
users can simply raise a determined.InvalidHP exception in their model code when the trial is
first created in its constructor or at any subsequent point during training. This user-raised
exception is then handled by Determined’s system internally – resulting in the graceful stop of the
current trial being trained, logging the InvalidHP exception in the trial logs, and propagating that
information to the search method.
Warning
It is important to note that each search method has different behavior when a
determined.InvalidHP is raised by the user in accordance with the internal dynamics of each
searcher, as detailed below.
HP Search Constraints in PyTorch vs. TF Keras/Estimator¶
Since the PyTorch and TF Keras/Estimator APIs have different behavior, the timing/placement of user-raised InvalidHP exceptions are somewhat different.
In the case of PyTorch, this exception can be raised in the trial’s __init__, train_batch,
or evaluate_batch methods. In the case of either TF Keras or TF Estimator, this exception can be
raised in the __init__ method or in an on_checkpoint_end callback.
See the hp_constraints_mnist_pytorch example for a demonstration of HP Search Constraints with PyTorch.
Searcher-Specific Behavior for HP Search Constraints¶
Search Algorithm |
HP Search Constraints Behavior |
|---|---|
Single |
Not applicable to HP Search Constraints as only a single hyperparameter configuration will be trained. |
Grid |
Does nothing since grid does not take actions based on search status or progress. |
Random |
Gracefully terminates current trial, creates a new trial with a randomly sampled set of hyperparameters and adds it to the trial queue. |
Adaptive (ASHA) |
Gracefully terminates and removes metrics associated with the current trial and creates a new trial with a randomly sampled set of hyperparameters. |