C2 - sweep jobs & hyperparameter tuning

a sweep job repeatedly executes a training job using hyperparameters selected from a given search space

the search space is a set of all allowed values

Azure ML supports expressions such as discrete choices, integer ranges, and several continuous distributions

discrete choices, eg: max_depth = choice(4,8,14,16, so each trial receives one of these values

continuous distributions, eg: learning rate search space is ${10}^{-4}\le \eta \le {10}^{-1}$

not every possible float needs to be enumerated

sample from a distribution

log scales are useful, eg: ${10}^{-3},{10}^{-2},0.1$

if two hyperparameter search spaces are given, a grid search evaluates combinations from the cartesian product

for a large number of hyperparameters, the curse of dimensionality shows up, eg: 4 hyperparameters with 10 values each would give ${10}^4$ trials

they can become expensive very quickly

tandom search samples random configurations rather than testing every configuration

Azure ML's current sweep schema supports random sampling, including seed and a Sobol quasi-random option

for high-dimensional spaces under limited budgets, random search can be surprisingly effective

the con is that it ignores previous results, rather than focusing around promising configurations

Bayesian sampling uses previous observations to guide future trials

a primary metric is optimised by the sweep

the optimisation goal could be to maximize/minimize

eg: primary_metric = 'f1' / goal = maximize

it does not replace quality gates

Azure ML sweep jobs expose limits including max_total_trials, and can also limit total sweep duration/trial duration

Azure ML sweep configuration supports limiting concurrent trials separately from total trials

max concurrent trials is not the same as max nodes

sweep parallelism also doesn't turn a trial into distributed training, where one trial is split into different nodes

Azure ML currently supports Bandit, Median Stopping and Truncation Selection policies for sweep early termination

a Bandit policy compares trial performance against the best-performing trial using an allowed slack, eg: best trial: F1 = .90, allowed slack = 0.10, if a trials f1 = .60, then it gets terminated

exact policy can use an absolute slack or a proportional slack factor, along with evaluation intervals and a delay before evaluation begins

a median stopping policy compares a trial's progress with the median performance of other trials at comparable stages

this eliminates underperforming trails without requiring them all to finish

a truncation selection terminates some specified bottom percentage of poorly performing trials at evaluation intervals

eg: terminate the bottom 20% (4 trails) out of 20 active ones

the Azure ML policy exposes a truncation_percentage for this purpose

early termination should be delayed as a run might simply be slower to converge

as such, an eventual winner might be killed

so early termination policies can include a delay before evaluation starts

there is a trade-off between compute savings and prematurely killing promising trials

whether early termination is useful depends on:

• algorithm
• framework
• what intermediate metrics can be reported
• training duration

each sweep trial can log parameters, metrics and artifacts through MLflow

eg: a sweep job

SEARCH SPACE

max_depth:
choice(4, 6, 8, 10, 12)

learning_rate:
log-distributed between
0.001 and 0.1

n_estimators:
choice(100, 250, 500)

SAMPLING
random

OBJECTIVE
primary_metric = F1
goal = maximize

LIMITS
max_total_trials = 20
max_concurrent_trials = 4