Hyperparameter Search Constraints

Determined’s Hyperparameter Search Constraints API enables finer-grained control over the hyperparameter search space through the specification of additional constraints. Among other things, this functionality is particularly useful for incorporating prior knowledge/domain expertise into a hyperparameter search and constraining the search to models that fit a particular deployment environment.

Using Determined’s HP Search Constraints requires no changes to the configuration files. Rather, users can simply raise a determined.InvalidHP exception in their model code when the trial is first created in the __init__ or at any subsequent point during training. This user-raised exception is then handled by Determined’s system internally – resulting in the graceful stop of the current trial being trained, logging the InvalidHP in the trial logs, and propagating that information to the search method.


It is important to note that each search method has different behavior when a determined.InvalidHP is raised by the user in accordance with the internal dynamics of each searcher, as detailed below.

HP Search Constraints usage in PyTorch vs. TF Keras/Estimator

Since the PyTorch and TF Keras/Estimator APIs have different behavior, it follows that the timing/placement of user-raised InvalidHP exceptions would be different.

In the case of PyTorch, this exception can be raised in either the trial __init__, train_batch, or evaluate_batch. In the case of either TF Keras and TF Estimator, it is valid for the user to raise this exception in either the __init__ or in an on_checkpoint_end callback.

See this https://github.com/determined-ai/determined/tree/master/examples/features/hp_constraints_mnist_pytorch for an example on HP Search Constraints usage.

Searcher-Specific Behavior for HP Search Constraints

Search Algorithm

HP Search Constraints Behavior


Not applicable to HP Search Constraints as only a single hyperparameter configuration will be trained.


Does nothing since grid does not take actions based on search status or progress.


Gracefully terminates current trial, creates a new trial with a randomly sampled set of hyperparameters and adds it to the trial queue.

Adaptive (ASHA)

Gracefully terminates and removes associated metrics with the current trial and creates a new trial with a randomly sampled set of hyperparameters.