Hyperparameter Tuning

Assumes you know from ML-101

This lesson builds on ML-101 · Lesson 9 (Model Evaluation) and ML-101 · Lesson 11 (Gradient Descent & Optimization). You should already know what a validation set is, what k-fold cross-validation does, and why you would split off a held-out set before touching hyperparameters. You should remember that training a model is itself a fitting procedure whose hyperparameters — learning rate, regularisation strength, tree depth — have to be chosen by an outer search.

You should also remember from ML-101 · Lesson 10 (Overfitting, Bias & Variance) that comparing models on the training set is meaningless because every model can be made to fit the training set arbitrarily well. The held-out set is the only honest comparison, but if you reuse it for both model selection and final evaluation, you have used it twice and lost the property you wanted. The nested cross-validation we cover here is the disciplined answer to that problem.

Learning Objectives

1. Distinguish model parameters (learned from data) from hyperparameters (set before fitting) and articulate why hyperparameters require a separate search procedure.
2. Compute the effective budget of a hyperparameter search in terms of equivalent full-training runs, and explain why this is the right cost unit.
3. Implement successive halving: assign a budget to each configuration, evaluate, keep the top half, double the budget, repeat.
4. Implement Hyperband: a wrapper around successive halving that brackets the optimal initial budget by varying it over a geometric schedule.
5. Set up nested cross-validation so that the outer fold estimates the generalization error of the entire selection procedure, not just the best-found configuration.

Parameters vs hyperparameters

A parameter is a number that the learning algorithm adjusts using the training data: the coefficients of a linear regression, the weights of a neural network, the split points of a decision tree. A hyperparameter is a number that the algorithm does not adjust: the learning rate, the regularisation strength, the depth of the tree, the number of leaves, the choice of optimizer. Hyperparameters are chosen by an outer search over candidate configurations, each evaluated by training the model and measuring its performance on a held-out set.

The reason hyperparameters exist is that the model class is a family of functions indexed by hyper-parameters, and the algorithm needs to be told which family member to fit. There is no algorithm that can derive the optimal learning rate from the data alone — a learning rate that is good for a 7-billion-parameter transformer is bad for a logistic regression, and vice versa. This is why hyperparameter search is a separate procedure from training.

The simplest search is to try every combination on a grid: learning rates in $\{10^{-1}, 10^{-2}, 10^{-3}, 10^{-4}\}$, regularisation strengths in $\{10^{-3}, 10^{-2}, 10^{-1}, 1, 10\}$, depths in $\{3, 5, 7, 9\}$. The total number of configurations is the product of the grid sizes. For each configuration, you train the model and evaluate on a validation set; the configuration with the best validation score wins. This is grid search, and it is correct but wasteful: a budget of 80 evaluations is needed for the above grid, regardless of whether the answer turns out to be near a grid point or in the middle of the space.

The tuning budget, defined properly

The right unit for a tuning budget is epochs of equivalent full-training runs, not configurations. The reason: training a single configuration for 10 epochs costs about a tenth of training a single configuration for 100 epochs, but if the shorter run would have identified a bad configuration in 10 epochs anyway, the savings are real. Conversely, training a configuration for 100 epochs when 10 would have sufficed is wasted compute.

Concretely, let $B$ be the total budget in epoch-equivalent units. A grid search with $G$ configurations and $E$ epochs each costs $G \cdot E$ units. A random search with $R$ configurations and the same $E$ epochs costs $R \cdot E$ units. Successive halving and Bayesian optimisation try to spend the same $B$ units more efficiently by allocating more epochs to promising configurations and fewer to unpromising ones.

The budget $B$ is the constraint that drives the design of every tuning algorithm. If $B$ is small (say $B = 10$), only a handful of configurations can be tried, and the algorithm should spend most of the budget on a small number of high-quality configurations evaluated to convergence. If $B$ is large (say $B = 1000$), the algorithm can afford a broad initial scan followed by refinement of the best candidates. Successive halving and Hyperband are designed around this scaling: they have explicit knobs for $B$ and produce schedules that adapt to it.

Grid search and its cost

Grid search is the right choice when the hyperparameter space is small (two or three dimensions) and the interactions between hyperparameters are predictable. Its weakness is that it samples the space at fixed points and misses everything between them. If the optimal learning rate is $3 \times 10^{-3}$ and the grid uses $\{10^{-2}, 10^{-3}\}$, the optimum falls between grid points and is never evaluated.

The cost of grid search is also predictable: $G$ configurations, each trained for $E$ epochs, total cost $G \cdot E$. This is useful for planning — you can compute the cost before launching the search — but it is also the source of the inefficiency. The configuration that will turn out to be best receives the same $E$ epochs as every other configuration, including those that are obviously bad from the start.

Random search as a baseline

Bergstra & Bengio (2012) argued that random search is competitive with grid search when some hyperparameters matter much more than others. The intuition: grid search allocates evaluations to the product of grid sizes, so changing a grid point on an unimportant hyperparameter wastes an evaluation that could have been spent on an important one. Random search samples each hyperparameter independently, so it spends evaluations on the important dimensions at the rate they deserve.

For a budget of $R$ random configurations, the expected minimum distance from the sampled point to the true optimum decreases as $R^{-1/d}$ in $d$ dimensions, but the constant is better for random search than grid search when the function depends on only a few of the $d$ dimensions. In practice, random search with 60 evaluations often beats grid search with 600 evaluations when the search space is wide and most hyperparameters are not influential.

import numpy as np
from sklearn.model_selection import RandomizedSearchCV

# A random search over a wide space with 60 evaluations.
# The budget here is 60 * training_time_per_config; tune that explicitly
# by setting n_iter to a value consistent with your compute budget.
param_distributions = {
    "learning_rate": np.logspace(-4, -1, num=200),
    "weight_decay":  np.logspace(-4,  0, num=200),
    "depth":         [3, 5, 7, 9, 12, 16],
    "min_child_weight": np.logspace(-2, 2, num=100),
}
search = RandomizedSearchCV(
    estimator=model,
    param_distributions=param_distributions,
    n_iter=60,
    scoring="neg_log_loss",
    cv=5,
    random_state=0,
)
search.fit(X_train, y_train)

Random search is the baseline against which more sophisticated methods are judged. If a fancier method does not beat random search on your problem, do not deploy it — random search is essentially free and the sophistication has a maintenance cost.

Successive halving

Successive halving is a budget-allocation algorithm that spends most of the budget on configurations that look promising after a short evaluation. The procedure is:

1. Sample $n$ configurations from the search space.
2. Allocate each a small initial budget $r_0$ (e.g. $r_0 = 1$ epoch or $r_0 = 32$ training examples).
3. Evaluate all $n$ configurations on the initial budget.
4. Keep the top $1/\eta$ fraction (typically $\eta = 3$ or $\eta = 4$), discard the rest.
5. Double the budget per configuration and repeat.
6. Stop when only one configuration remains or when the budget per configuration reaches a maximum.

The keep-fraction $\eta$ is the halving rate. The initial number of configurations $n$ and the initial budget $r_0$ are chosen so that the total budget $B = n \cdot r_0 + (n/\eta) \cdot r_0 \cdot \eta + \ldots$ is approximately fixed. Successive halving's strength is its adaptivity: it does not need to know in advance which configurations will be best, and it concentrates compute on those that prove themselves early. Its weakness is that it requires a minimum budget $r_0$ that is large enough to discriminate good from bad configurations — if $r_0$ is too small, the early elimination is random noise.

def successive_halving(configurations, evaluate, budget, eta=4, min_budget=1):
    """Allocate a fixed total `budget` across configurations by successive halving.

    `evaluate(config, b)` returns the validation score for `config` at budget `b`.
    The function returns the configuration with the highest final-budget score
    and the full trace of budgets and scores for inspection.
    """
    configs = list(configurations)
    r = min_budget
    while len(configs) > 1:
        # Evaluate all surviving configurations at the current budget.
        scores = [evaluate(c, r) for c in configs]
        # Keep the top 1/eta fraction.
        n_keep = max(1, len(configs) // eta)
        ranked = sorted(zip(scores, configs), reverse=True)
        configs = [c for _, c in ranked[:n_keep]]
        r *= eta   # double the budget for the survivors
    return configs[0]

The choice of $\eta$ is a budget-versus-precision knob. Larger $\eta$ means more aggressive pruning (fewer survivors each round) and more savings on bad configurations, but also a higher risk of discarding a configuration that would have been best given more budget. Smaller $\eta$ means more configurations survive each round, more thorough evaluation, but less compute savings.

Hyperband: bracketing the budget

Hyperband (Li et al., 2017) wraps successive halving with a bracket structure that varies the number of initial configurations and the initial budget. The intuition: for a fixed total budget $B$, there is a tradeoff between the number of configurations you can afford to start with (wide initial scan, shallow evaluation) and the depth of evaluation you can afford per surviving configuration (narrow initial scan, deep evaluation). Neither extreme is uniformly best, so Hyperband runs multiple brackets, each with a different point on this tradeoff.

A Hyperband bracket is parameterised by two integers: $R$, the maximum budget per configuration in powers of $\eta$, and $s$, the budget index of the current bracket ($0 \le s \le R$, where $s = R$ is the most aggressive pruning and $s = 0$ is the most thorough evaluation). The bracket proceeds as follows:

• Start with $n = \lceil B / r_0 \cdot \eta^{-s} \rceil$ configurations and budget $r_0 \cdot \eta^s$ per configuration.
• Apply successive halving within the bracket until only one configuration remains or the maximum budget is reached.

The collection of brackets, run in sequence, covers the entire tradeoff curve. The total budget across all brackets is approximately $B \cdot (R + 1)$, which is the cost of running Hyperband with maximum budget ratio $R$ instead of a single successive-halving run with a fixed budget split.

The key Hyperband invariant: each bracket spends the same total budget, but on a different mix of breadth (number of configurations) versus depth (budget per configuration). The bracket with $s = R$ starts with many configurations and prunes aggressively, spending most of the budget on identifying which ones survive. The bracket with $s = 0$ starts with few configurations and evaluates them deeply, trusting the initial random sample to include a good configuration.

The practitioner-facing version of this is: the rung rule. In successive halving, the $k$-th rung of the ladder requires the configurations that survive to the $k$-th round to have been trained for $r_0 \cdot \eta^k$ steps. Hyperband's contribution is to choose how many rungs to give each configuration by varying the initial bracket.

Bayesian optimisation in one paragraph

Bayesian optimisation models the validation score as a sample from a Gaussian process over the hyperparameter space and chooses the next configuration to evaluate by maximising an acquisition function (expected improvement, upper confidence bound, etc.) over the posterior. It is the right tool when the cost of evaluating one configuration is high and the number of evaluations is small (less than a few hundred). For deep learning, where a single evaluation might take hours, Bayesian optimisation with a Tree-structured Parzen Estimator (TPE) is the default in libraries like Optuna and Hyperopt. For classical ML with fast training, random search or Hyperband is competitive and simpler.

The reason we do not cover Bayesian optimisation in detail here is that the value it adds over random search on most real problems is smaller than the variance across runs: random search with 200 evaluations typically matches Bayesian optimisation with 50 evaluations on a wide search space, and the engineering cost of random search is lower. When you have evidence that you need smarter search (a long-running training job where each evaluation costs hours and you can afford only 30 evaluations), switch to Bayesian optimisation; otherwise stay with random search or Hyperband.

Nested cross-validation

The final subtlety: when you use cross-validation to select hyperparameters, the cross-validated score is biased. It is biased because you selected the hyperparameters that looked best on the validation folds, and the validation folds were used both for selection and for evaluation. The held-out score you report is therefore optimistically biased.

The fix is nested cross-validation. An outer loop splits the data into $K$ outer folds. For each outer fold, the outer training set is itself split into $L$ inner folds, and hyperparameter selection is performed by $L$-fold cross-validation on the outer training set. The best hyperparameters from the inner loop are then refit on the entire outer training set and evaluated on the outer test fold. The $K$ outer test scores are averaged to give an unbiased estimate of the generalization error of the entire selection procedure.

The cost is $K \times L$ fits per configuration, plus a final refit on each outer fold. For $K = L = 5$, that is 25 fits per configuration. This is the cost of doing hyperparameter selection honestly.

from sklearn.model_selection import GridSearchCV, KFold

# Nested CV: outer loop gives an unbiased estimate of the selection procedure.
outer = KFold(n_splits=5, shuffle=True, random_state=0)
inner = KFold(n_splits=4, shuffle=True, random_state=1)
search = GridSearchCV(estimator=model, param_grid=grid, cv=inner)

outer_scores = []
for tr_idx, te_idx in outer.split(X):
    search.fit(X[tr_idx], y[tr_idx])
    score = search.score(X[te_idx], y[te_idx])
    outer_scores.append(score)
# Mean of outer_scores is the unbiased estimate of generalization error
# of the *selection procedure*, not just of the best-found configuration.

A common mistake is to report the inner cross-validation score as the final generalization estimate. The inner score is the score of the best configuration as estimated by the inner procedure, which is itself optimistic. The outer loop is the only honest measurement.

Conditional hyperparameters and search-space design

A search space is rarely a Cartesian product of independent axes. The right choice for n_estimators in a gradient boosting model depends on learning_rate (slow learners need more rounds); the right choice for min_child_weight depends on the dataset size; the right weight_decay for AdamW depends on the model's parameter count. Failing to encode these dependencies makes the search grid sample configurations that are obviously bad — n_estimators = 50 with learning_rate = 0.3 is a known failure mode — and burns budget on them.

The remedy is conditional hyperparameters: the search space is a tree of distributions, where the range of one hyperparameter depends on the values of others. Optuna and Ray Tune both support this natively. Concretely, in Optuna you write trial.suggest_float("weight_decay", 1e-4, 1e-2, log=True) and then n_estimators = trial.suggest_int("n_estimators", 50, 2000) with the understanding that lower learning rates need more rounds. Search spaces that respect these relationships have a much higher chance of finding strong configurations in a fixed budget.

import optuna

def objective(trial, X, y):
    # Conditional search space: learning rate and number of trees are linked.
    lr = trial.suggest_float("learning_rate", 1e-3, 1e-1, log=True)
    # Higher learning rate -> fewer rounds needed; cap accordingly.
    n_est = trial.suggest_int(
        "n_estimators", 100, 5000,
        # Use a soft cap via the suggestion range; finer tuning is wasteful.
    )
    max_depth = trial.suggest_int("max_depth", 3, 12)
    min_child_weight = trial.suggest_float(
        "min_child_weight", 1e-2, 1e2, log=True,
    )
    model = GradientBoostingClassifier(
        learning_rate=lr, n_estimators=n_est,
        max_depth=max_depth, min_child_weight=min_child_weight,
    )
    score = cross_val_score(model, X, y, cv=4, scoring="neg_log_loss").mean()
    return score

study = optuna.create_study(direction="maximize")
study.optimize(lambda t: objective(t, X, y), n_trials=200, timeout=3600)

A second design choice is the sampling distribution. Learning rates, regularisation strengths, and weight-decay coefficients are usually sampled on a log-uniform distribution (equal probability per decade) because their effect is multiplicative — a change from $10^{-3}$ to $10^{-2}$ has the same qualitative impact as a change from $10^{-2}$ to $10^{-1}$. Sampling uniformly in $[0, 1]$ would assign most of the probability to large learning rates that diverge the training. Counts (number of trees, batch size, layer width) are usually sampled uniformly in linear space.

Log-uniform sampling and the order-of-magnitude argument

A learning rate $\eta$ that works well at $10^{-3}$ is too small at $10^{-1}$; a learning rate that works well at $10^{-1}$ is too large at $10^{1}$. The relevant axis is the order of magnitude, not the linear value. Sampling uniformly on the log axis — drawing $u \sim \text{Uniform}(\log a, \log b)$ and returning $e^u$ — places equal probability on each decade. The expected log-distance from the sample to the true optimum is then $(b - a)^{-1}$ in log-space, which is small when the search interval is well-chosen.

The mistake this avoids: sampling $\eta \sim \text{Uniform}(0, 1)$ and dividing by 100. That puts most of the probability near $\eta = 0$, which is exactly the regime where training stalls. The 101-level framing is "try a few values and pick the best"; the 201-level framing is "draw samples on the right axis so the expected distance to the optimum is small".

This is also why random search's empirical advantage over grid search depends on the search-space axis. Grid search with points $\{10^{-1}, 10^{-2}, 10^{-3}\}$ samples the log axis uniformly (good); grid search with points $\{0.1, 0.3, 0.5\}$ samples the linear axis uniformly and misses the regime below $10^{-1}$ entirely (bad). When a colleague shows you a grid-search result that "didn't find anything below a certain threshold", the diagnosis is usually the wrong axis, not the wrong algorithm.

Worked example: Hyperband brackets by hand

To make Hyperband concrete, work out a single bracket with $R = 3$, $\eta = 4$. The bracket starts with $n = 16$ configurations and initial budget $r_0 = 1$ epoch. The successive-halving rounds have budgets $r_0, \eta r_0, \eta^2 r_0, \eta^3 r_0 = 1, 4, 16, 64$ and the number of survivors is $n, n/\eta, n/\eta^2, n/\eta^3 = 16, 4, 1, 0.25 \to 1$. Total budget used by this bracket is

$B_{\text{bracket}} \;=\; \sum_{k=0}^{R-1} \frac{n}{\eta^k} \cdot \eta^k r_0 \;=\; n \cdot r_0 \cdot R \;=\; 16 \cdot 1 \cdot 3 \;=\; 48$

epoch-equivalents. The next bracket, with $s = R - 1$, halves the initial configurations and doubles the initial budget: $n' = n / \eta = 4$, $r_0' = \eta r_0 = 4$. Running all $R + 1 = 4$ brackets, the total budget is approximately $B_{\text{bracket}} \cdot (R + 1) = 48 \cdot 4 = 192$ epoch-equivalents.

This calculation is the practitioner's planning tool. When a manager asks "can we afford Hyperband?", the answer is "yes, if you can spend $B \cdot (R + 1)$ epoch-equivalents". When the budget is small, use fewer brackets (smaller $R$); when the budget is large, use more brackets to cover more of the breadth-versus-depth tradeoff.

The bracket $s = 0$ is effectively a single configuration trained to the maximum budget — Hyperband degenerates to "run a few configurations for a long time". The bracket $s = R$ is effectively a large random search with aggressive pruning. The brackets in between trade off the two regimes.

Population-based training and meta-search

All the algorithms above evaluate each configuration independently and pick the best. Population-based training (PBT) instead maintains a population of configurations, evaluates them in parallel, and at intervals perturbs poorly-performing members of the population with information from well-performing members. The procedure is roughly: (1) train the population for $k$ steps; (2) score each member on a validation set; (3) replace the worst-performing member's weights with the best-performing member's weights (a "exploit" step); (4) perturb the hyperparameters of the replaced member with noise (an "explore" step); (5) continue training.

PBT is more expensive per step than random search (it maintains a population rather than a single configuration), but it discovers schedules of hyperparameters rather than fixed values. The output of a PBT run is not "the best learning rate was $3 \times 10^{-4}$" but "the learning rate should be $10^{-3}$ for the first $1000$ steps, then annealed to $10^{-4}$ for the next $5000$ steps". This is closer to how practitioners actually use learning rate schedules, and it is one of the reasons PBT is the standard for tuning large-scale neural network training.

The cost is $P \times \text{(cost per config)}$, where $P$ is the population size (often $20$ to $50$). For $P = 32$ and a model that takes an hour to train, PBT can fill a small cluster. For cheaper models (linear, gradient boosting) the cost is comparable to random search and PBT adds little value. The right tool depends on the cost per evaluation and the expected benefit from discovering schedules.

When NOT to tune (ablation discipline)

The final, often-omitted point: not every knob needs to be tuned. Most of a model's performance comes from a small number of high-leverage hyperparameters (regularisation strength, learning rate, capacity), and the rest can be left at library defaults. Ablation discipline is the practice of running controlled experiments to determine whether a hyperparameter matters before spending budget on it.

A simple ablation procedure: take the candidate hyperparameter, run the model at its default value and at a perturbed value, and compare the validation score. If the gap is within the noise of cross-validation (a few standard errors of the cross-validated mean), the hyperparameter does not matter for this problem and should be left at the default. If the gap is large, the hyperparameter matters and is worth tuning.

The ablations to run, in order of expected leverage:

1. Regularisation strength ($\lambda$ or weight_decay). Almost always matters. Default values are usually wrong for your specific problem.
2. Learning rate ($\eta$). Matters whenever the optimizer is Adam or AdamW; matters less for SGD-with-momentum with cosine annealing, where the schedule has its own dynamics.
3. Capacity (depth, width, number of trees). Matters when the gap between training and validation loss is large; matters less when the model is already capacity-limited.
4. Batch size. Matters for optimizers with momentum; less so for Adam.
5. Everything else: dropout, label smoothing, scheduler details, kernel sizes. Rarely matters by more than a fraction of a percent. Leave at defaults unless ablations suggest otherwise.

The wrong move is to launch a 1000-trial Hyperband sweep over 12 hyperparameters without first determining that those 12 hyperparameters matter. The right move is to ablate the top three first, then sweep the survivors.

def ablation_study(model_fn, X, y, hp_name, hp_values, n_folds=4):
    """Run a one-factor-at-a-time ablation for a single hyperparameter.

    Returns a dict {hp_value: cross-validated score}. Use the variance
    across folds to determine whether the gap between values is signal
    or noise.
    """
    scores = {}
    for v in hp_values:
        model = model_fn(**{hp_name: v})
        cv_scores = cross_val_score(model, X, y, cv=n_folds, scoring="neg_log_loss")
        scores[v] = (cv_scores.mean(), cv_scores.std())
    return scores

The result of an ablation is a small set of "matters" hyperparameters and a larger set of "doesn't matter" hyperparameters. The latter are set to library defaults and left alone; the former enter the search space and get the tuning budget. This split is the artefact the tuning run should produce, alongside the best configuration. A sweep that returns only a configuration — without a record of which other hyperparameters were tested and confirmed irrelevant — cannot be reproduced by a colleague who suspects the result was due to an untested factor.

A subtler point: the ablation must control for everything else. If you change the learning rate and the schedule at the same time, you cannot tell which change produced the improvement. The ablation discipline is single-factor-at-a-time, with all other factors held at the values used in the previous experiment. This is the same logic as the scientific method, and it is the reason most "we improved X by changing Y" papers are not reproducible.

Tracking the tuning process and stopping early

Tuning is a sequential decision problem: after every $k$ trials, you must decide whether to stop (you have found a strong enough configuration) or continue (the budget is not yet exhausted and there is more to gain). The naive stopping rule is "stop when the budget is exhausted"; the smarter rule is to estimate the expected value of continuing and stop when that estimate falls below a threshold.

A practical heuristic is the marginal improvement rule. After each batch of $k$ trials, compute the improvement in the best-so-far validation score. If the improvement over the last $k$ trials is below a threshold (e.g. $10^{-3}$ in log-loss for a binary task), stop. This rule is sensitive to the choice of $k$ and the threshold, but it is much better than running every trial to budget exhaustion.

def marginal_improvement_stopper(scores, k=10, tol=1e-3):
    """Return True if the marginal improvement over the last k trials is below tol.

    `scores` is the list of validation scores seen so far, sorted by trial
    order. The function looks at the best score in each of the last k
    trials and stops when the slope flattens.
    """
    if len(scores) < k:
        return False
    recent = scores[-k:]
    return (max(recent) - min(recent)) < tol

A more principled approach uses a Bayesian model of the validation score as a function of the hyperparameter configuration. The model predicts, with uncertainty, the score at any untried configuration. The expected improvement from running one more trial is the expectation over the posterior of $\max(0, y^* - y_{\text{new}})$, where $y^*$ is the current best. When the expected improvement falls below the cost of one trial, stop. This is the Bayesian optimisation stopping rule and is implemented in libraries like Optuna.

The risk of stopping too early: the configuration that would have been best lies just outside the explored region, and stopping now returns a configuration that is good but not great. The risk of stopping too late: every additional trial returns a configuration that is within the noise of the current best, and the compute is wasted. The right balance depends on the cost per trial and the expected improvement from a longer search.

Search-space leakage and reproducibility

A subtle form of search-space leakage: the search space itself encodes assumptions about the problem that may not hold. If the search space excludes certain regions (e.g. learning_rate > 0.1 is clipped to 0.1), the algorithm will never find configurations that lie outside. If the data distribution shifts between the tuning and production data, the optimal hyperparameters may shift too.

The reproducibility point is sharper. A hyperparameter search that uses random sampling without a fixed seed is non-reproducible: the same code run twice produces different trials, and the reported best configuration depends on the seed. The fix is mechanical: pass random_state to every search algorithm that accepts it, log every configuration and every score, and store the fitted model object together with the configuration that produced it. The pipeline + configuration + seed is the artefact you ship, not just the model weights.

import optuna

# Pinning the seed for reproducibility.
sampler = optuna.samplers.TPESampler(seed=42)
study = optuna.create_study(direction="maximize", sampler=sampler)
study.optimize(objective, n_trials=200)

# Log the full history.
for trial in study.trials:
    print(trial.number, trial.params, trial.value)

A subtler reproducibility trap is the version drift of the search library itself. Optuna's TPE sampler has changed its default behaviour across versions; the same code with a pinned optuna==3.1.0 versus optuna==3.6.0 may produce different trials for the same seed. Pin the library version in the experiment log, just as you would pin the model and dataset versions.

Tuning in production: drift and re-tuning schedules

In production, the data distribution drifts over time: a model tuned on January's data may be suboptimal on July's data. The naive response is to re-tune every month. The 201-level response is to recognise that most hyperparameters drift slowly and that re-tuning is expensive.

The right cadence depends on the cost of a tuning run and the rate of distribution shift. A model serving ads can tolerate a $0.1\%$ AUC drop and should be re-tuned every few weeks; a model underwriting insurance cannot tolerate any drop and should be re-tuned continuously with shadow deployments. The middle ground — re-tune when a monitored metric drops by more than a threshold — is the standard pattern.

A second consideration is the configuration space itself. If the optimal hyperparameters drift slowly, a narrow search around the current best is enough; if the optimum can jump to a different region (e.g. the dataset grows past a threshold where a different architecture is preferable), a broader search is needed. The right choice depends on whether you expect the model class to remain the best choice or whether the data is evolving past it.

A third consideration is the coupling between tuning cadence and downstream monitoring. If the production monitoring detects drift before the next scheduled re-tune, the system has to either pause predictions or accept the suboptimal configuration until the re-tune completes. A continuous tuning pipeline that responds to drift signals is the modern answer: when the monitor flags a regression, the pipeline launches a fresh sweep and promotes the winner only if it improves on the current configuration by more than the promotion threshold. The right cadence is the one that keeps the average performance within the service-level objective without over-spending on tuning compute.

def shadow_compare(current_model, new_model, X_recent, y_recent, threshold=0.005):
    """Compare the current production model to a candidate on the most recent data.

    Returns True if the new model should be promoted. The threshold is the
    minimum improvement in log-loss required for promotion; a model that
    improves by less than the threshold is not worth the operational cost
    of a new deployment.
    """
    current_score = log_loss(y_recent, current_model.predict_proba(X_recent))
    new_score = log_loss(y_recent, new_model.predict_proba(X_recent))
    return (current_score - new_score) > threshold

The cost of a re-tune is usually dominated by the cost of the best configuration's full training run, not by the search itself. A Hyperband sweep over 200 configurations with $\eta = 4$ uses a total of approximately $B \cdot (R + 1)$ epoch-equivalents; if the best configuration is then trained for the full budget, the sweep's cost is amortised over many production predictions. This is why "tune once and deploy for a year" is sometimes the right call, especially when the data distribution is stable.

A final production concern is warm-starting the search. The history of past tuning runs is informative: a configuration that was best on last month's data is a good prior for the current search. Libraries like Optuna support warm-starting by loading the study from storage and continuing the optimisation. The prior is not always right (the optimum may have shifted), but it is much better than sampling from scratch.

Common tuning mistakes and how to read the diagnostics

A tuning run that produces no improvement over the default is rarely the algorithm's fault. The most common failure modes are diagnosable from the trial history.

If the best configuration's score is close to the median configuration's score, the hyperparameters you are tuning do not matter for this problem — the model is dominated by other factors (data quality, feature engineering, model class). The remedy is to ablate and identify which factors actually matter before tuning.

If the best configuration lies on the boundary of the search space, the true optimum is outside and the algorithm is wasting compute on a half-explored region. The remedy is to expand the search space in the direction of the boundary and re-run.

If successive halving or Hyperband keeps the wrong configurations (a configuration that scored well at the smallest budget but poorly at the largest budget), the early-evaluation metric is not predictive of the final score. The remedy is to use a more informative early-evaluation metric or to increase the initial budget $r_0$ so the early signal is more reliable.

If the cross-validated score has high variance across folds (the standard deviation across folds is comparable to the gap between configurations), the dataset is too small for the number of hyperparameters being tuned. The remedy is to gather more data, reduce the number of hyperparameters, or use a higher $K$ for the cross-validation (which averages out the fold-level variance).

def diagnose_trial_history(study, n_top=5):
    """Print a diagnostic summary of an Optuna study.

    Checks for the four common failure modes: collapsed scores,
    boundary optima, non-predictive early metrics, and high fold variance.
    """
    trials = sorted(study.trials, key=lambda t: -(t.value or -1))
    values = [t.value for t in study.trials if t.value is not None]

    if not values:
        print("No completed trials.")
        return

    median = sorted(values)[len(values) // 2]
    best = max(values)
    print(f"best={best:.4f}  median={median:.4f}  gap={best - median:.4f}")
    if best - median < 0.005:
        print("  WARN: best close to median -> hyperparameters may not matter")

    for t in trials[:n_top]:
        for k, v in t.params.items():
            if isinstance(v, (int, float)) and (v <= 1e-5 or v >= 1e5):
                print(f"  WARN: trial {t.number} has {k}={v} at a boundary")

The diagnostic routine is small but pays for itself the first time a tuning run returns an unexpected result. A configuration that is best on the validation set but degrades on the test set is one of the most common surprises, and the cause is almost always a leak in the inner CV (the same fold used for selection and evaluation) rather than a failure of the optimiser.

Reading the diagnostics carefully before launching a 1000-trial sweep is the difference between a tuning run that produces actionable findings and one that returns the default configuration with extra compute. The 201-level practitioner treats tuning as an experiment with hypotheses, controls, and diagnostics — not as a black-box optimisation.

Key Takeaways

• Hyperparameters are not learned from data; they are chosen by an outer search. The cost of the search is measured in epoch-equivalent units, and the budget $B$ is the constraint that drives the design of every tuning algorithm. Tuning is an experiment, not a black-box optimisation.
• Random search is the baseline; sophisticated methods (Hyperband, Bayesian optimisation) should be evaluated against it before being adopted. On many problems, random search with enough evaluations matches fancier methods at lower engineering cost.
• Successive halving allocates budget adaptively: bad configurations are pruned early, promising ones get more compute. Hyperband brackets the optimal initial budget by varying it across brackets.
• Nested cross-validation is the only honest measurement of the generalization error of a selection procedure. Reporting the inner cross-validation score overestimates performance because the inner folds were used for both selection and evaluation.
• The fitted model object — including the fitted preprocessor, the selected hyperparameters, and the trained weights — is the artefact you ship. The pipeline + the configuration that produced it is what runs in production.
• Search-space design is half the work: condition hyperparameters on each other, sample learning-rate-like quantities on a log-uniform axis, and ablate before sweeping to confirm which knobs actually matter.
• In production, plan for distribution drift: warm-start tuning runs from prior studies, monitor the production metric against the tuning-time estimate, and re-tune on a cadence that matches the drift rate.