Did you find better hyperparameters—or a lucky seed?

Your hyperparameter sweep has a winner. One run sits at the top of the validation table, so you save its configuration and launch a longer training job. The next run scores lower. Another configuration now looks better.

A stochastic training run measures a recipe under one particular sequence of random choices. A favorable initialization or training order can help a configuration reach the top of the table. The more results you inspect, the more opportunities you have to select a favorable fluctuation. Research on overfitting in model selection explains why optimizing an estimated score can also overfit the selection procedure.

Our compute-budget tuning guide recommends reserving runs to confirm finalists. This article makes that step concrete: compare two recipes over the same seed panel, retain every result, and decide whether the evidence supports a change. The companion uses a small CPU classifier so you can practice the method without an LLM fine-tuning budget.

Separate the sources of variation

“Try another seed” is incomplete advice unless you identify what the seed changes.

The question here is: how do these recipes compare across training randomness on this fixed dataset and evaluation procedure? It is not an estimate of performance on every future dataset, customer population, or deployment environment.

For an LLM study, pin decoding settings and the evaluator as well as the training recipe. If you want to study generation or judge variation, give those repetitions separate identifiers. Mixing all randomness into one seed makes the source of a difference harder to explain.

Reserve the confirmation budget before searching

Choose the number of finalists, seed panel, training budget, primary metric, and unacceptable regressions before looking at confirmation results. Use fresh confirmation seeds that were not selected for favorable pilot outcomes. If an incumbent matters, keep it in the comparison.

For example, a budget of 20 equal-cost fits could allocate nine to exploration, ten to two finalists across five seeds, and one to training the frozen release recipe. This is an illustrative allocation, not a recommended universal ratio. Evaluation, failed attempts, and startup overhead also consume resources; an LLM study should budget those costs explicitly.

The local companion starts with a ten-fit comparison panel. Its two settings are teaching choices, not winners from an executed search. The Polyaxon workflow below expands this into six sampled configurations, each evaluated over five training seeds: 30 fits before any additional confirmation or final assessment. Five repetitions make the workflow easy to inspect; they are not a guarantee of adequate precision.

Do not keep adding seeds until the candidate you prefer appears to win. If the planned evidence is inconclusive, report that outcome and decide whether another, explicitly budgeted study is worth running.

Match the experiment, then pair the results

Both candidates should use the same training rows, preprocessing, validation rows, scorer, and training budget. Each fit starts from fresh model state. Changing those conditions between repetitions answers a different question.

Use a common seed panel and compare candidates within each seed. In the companion, a fresh random generator produces the same sequence of training permutations for candidate A and candidate B. Only the regularization setting changes. This gives the pairing a concrete meaning.

For each seed, compute:

paired difference = candidate B validation score − candidate A validation score

A positive value favors B for a higher-is-better metric. A negative value favors A. Retain the individual scores so that the difference can be traced back to both runs.

Matching integer seeds is not enough to guarantee matched stochastic behavior across unrelated implementations. Models with different architectures can consume random numbers differently. Where feasible, control data order and augmentation separately; describe what is actually shared rather than assuming every random event aligns.

Run a small confirmation panel

The hyperparameter-seeds companion compares two regularization settings for scikit-learn's SGDClassifier on its bundled digits dataset. No separate data download or GPU is required.

Using a constant learning rate is deliberate. With the estimator's optimal schedule, alpha also participates in the learning-rate calculation. Fixing eta0 under a constant schedule makes the changed setting's role easier to explain. The SGDClassifier reference documents those controls and the one-pass behavior of partial_fit.

The script reserves 20% of rows for final testing, then assigns 25% of the remaining development rows to validation. Both stratified splits use fixed split seeds. A scaler fits only the training rows; the test partition is never scored. Use the evaluation-data leakage guide to choose appropriate group or time boundaries for a real dataset; repeated training cannot repair a leaky split.

With Python 3.12 available:

git clone https://github.com/polyaxon/polyaxon-examples.git
cd polyaxon-examples/blog/hyperparameter-seeds
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python study.py --output outputs/seed-study

Use an empty output directory. The script refuses to overwrite an existing panel. It alternates candidate execution order between seed blocks and limits native thread pools to one, keeping this exercise straightforward to inspect.

The companion is source-reviewed but has not been executed for this article. No score table or winning setting is claimed here; the files created by your execution supply that evidence.

Read the full panel before choosing a recipe

Start with plan.json and split-manifest.json. They record settings, training seeds, split assignments, source and dataset hashes, and runtime versions. resolved-requirements.txt retains the installed packages.

Each trial-<candidate>-<seed>.json contains its status, completed training budget, validation macro F1, class report, and measured fit time. A matching prediction file retains validation row IDs, labels, and predictions. summary.json then reports:

• Each candidate's mean, sample standard deviation, minimum, and maximum validation score.
• The score difference for each matched seed, plus the mean and spread of those differences.
• How many pairs favor A, favor B, or tie exactly.
• Fit-time summaries, kept separate from the quality comparison.

Mean and spread answer different questions. The mean summarizes the observed panel; the standard deviation describes how much its training outcomes vary. Neither is a confidence interval for performance on future production traffic. A small observed spread also does not prove that rare unstable runs cannot occur.

For macro F1, every label contributes equally to the average. Inspect the per-class reports: a better overall mean can coexist with a regression on a class your application cannot afford to mishandle. This teaching dataset does not define that business constraint for you.

Define a practically meaningful improvement for your application before interpreting tiny differences. A positive mean difference can be too small to justify additional complexity or cost. Conversely, a consequential regression can matter even when an overall mean improves. The script deliberately leaves that decision to the study's declared requirements.

Keep failures and resource use in the comparison

A candidate that completes only its favorable repetitions should not be compared with another candidate's complete panel. The companion preserves ordinary trial exceptions as failed records. If any trial fails, it marks the summary incomplete, omits the finalist comparison, and exits nonzero.

A forced process kill can leave an unfinished record and no summary. Preserve that evidence, diagnose the cause, and run a documented replacement panel rather than silently dropping the missing result. For larger studies, retain retry attempts and distinguish infrastructure interruption from a configuration that consistently exceeds memory or runtime limits.

The example measures model construction and training time. That measurement excludes preprocessing, validation, file writes, dependency installation, and scheduling. It is not allocated GPU-hours or a billing estimate. Equal training passes make the learning budget comparable, but do not guarantee equal cost for arbitrary models.

If you change the final training duration, repeat the finalist comparison at the intended duration. A recipe that looks favorable after a short pilot need not keep the same ranking after a longer run.

Sample configurations with a random-search matrix

The companion's polyaxonfile.yaml uses a random-search matrix to sample six alpha values from eight choices. Each child operation evaluates one sampled configuration over the same five training seeds. That gives independent configuration-level metrics, artifacts, failures, and retries while keeping every seed panel complete.

Keep the training seed out of the random-search parameters. Sampling (alpha, seed) pairs independently could leave a configuration with one repetition and another with several, undermining the comparison. Here, matrix.seed: 19 controls configuration sampling; 11, 23, 37, 53, 71 control training order inside every child.

version: 1.1
kind: operation
name: hyperparameter-seed-random-search
matrix:
  kind: random
  numRuns: 6
  concurrency: 2
  seed: 19
  params:
    alpha:
      kind: choice
      value: [0.00001, 0.00003, 0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03]
component:
  inputs:
    - name: alpha
      type: float
  run:
    kind: job
    container:
      image: python:3.12-slim
      workingDir: "{{ globals.run_artifacts_path }}/uploads"
      command: [sh, -c]
      args:
        - |
          set -eu
          python -m pip install --no-cache-dir -r requirements.txt
          exec python study.py --tracked --alpha {{ alpha }}
      resources:
        requests:
          cpu: "1"
          memory: 1Gi

From the companion directory, with your CLI configured and an existing project:

polyaxon run -p quick-start -f polyaxonfile.yaml -u

Replace quick-start with your project. The -u flag uploads the local folder; the included .polyaxonignore excludes local environments and results. Requirements include polyaxon and install at startup. Workers need image and package-index access and a configured artifact store. Pin reviewed dependency versions and the base image digest for a reproducibility-sensitive study.

numRuns: 6 bounds the sampled child operations, while concurrency: 2 permits at most two children at once. Each child performs five sequential fits, so the complete search contains 30 fits. Retries add work; neither field imposes a monetary budget.

Each child logs sampled_validation_macro_f1 and sampled_fit_seconds at seed indices one through five. Those steps represent repetitions, not training progress. Once its panel is complete, it also logs validation_macro_f1_mean, validation_macro_f1_sd, and fit_seconds_mean. Use the comparison dashboard to compare these summaries alongside the sampled alpha, then inspect the individual scores and class reports. An incomplete panel fails and does not emit those aggregate metrics.

Every child writes reports under tracking.get_outputs_path()/seed-study before registering artifact references. Saving files and logging their lineage are separate responsibilities. This design gives one run per sampled configuration, not one run per seed; retrying a child repeats its entire panel.

Compare two completed child panels

After reviewing all six configurations, download the reports for two candidates using their child UUIDs, not the matrix parent's UUID. With the companion dependencies installed locally:

RUN_A="replace-with-first-child-uuid"
RUN_B="replace-with-second-child-uuid"
polyaxon ops artifacts -p quick-start -uid "$RUN_A" --dir outputs/seed-study --path-to outputs/child-a
polyaxon ops artifacts -p quick-start -uid "$RUN_B" --dir outputs/seed-study --path-to outputs/child-b
python compare_panels.py outputs/child-a outputs/child-b > outputs/paired-comparison.json

Use fresh download directories. The helper locates each extracted panel, checks its source, data, split, runtime, training settings, and seed list, and requires one completed trial per seed. It then reports B-minus-A differences using the actual sampled values. Missing, duplicate, failed, or incompatible records cannot silently become a successful comparison. The comparison JSON records the source plan paths and hashes.

These are still search results: selecting two configurations after inspecting six panels does not make their existing results independent confirmation. Freeze the shortlist, then run the selected recipes on a fresh, predeclared seed panel before final assessment. The local script accepts both settings and a replacement seed list. For example, replacing the two alpha values below with your actual finalists:

python study.py --alpha-a 0.0001 --alpha-b 0.01 --seeds 83 97 109 127 149 --output outputs/confirmation

That adds ten fits, bringing this search-plus-confirmation design to 40 fits before training and evaluating the frozen release artifact. The shown settings are illustrative, not claimed search winners. Repeated use of the validation set still remains part of model selection.

The children terminate after their panels complete. If you need to stop the search, use the parent UUID returned by submission:

polyaxon ops stop -p quick-start -uid YOUR_MATRIX_RUN_UUID

Keep the reports for review, then apply your normal local-output or run-artifact retention policy. The example creates no service or PVC.

Freeze the recipe before opening the final test

Repeating seeds reduces reliance on a single training outcome. It does not undo repeated adaptation to the same validation set. A confirmation panel remains part of model selection, and its scores should be reported as such.

Freeze the hyperparameters, training duration, preprocessing, checkpoint rule, and evaluation procedure before the final assessment. Also decide how the deployed artifact is produced: for example, a predeclared training seed or an explicitly budgeted ensemble. Choosing whichever confirmation seed has the highest validation score is another selection step, not proof that the recipe is robust.

The companion reserves test rows but intentionally provides no automatic test evaluation or model promotion. Evaluate the actual frozen release artifact separately. If you use that result to revise the recipe, it becomes development feedback; it is no longer untouched final evidence for the revised choice.

Final thoughts

The highest score in a sweep is a reason to investigate a configuration. A complete confirmation panel helps you decide whether its advantage persists across the training variation you care about.

Start with two finalists, a fixed seed panel, and a written decision rule. Keep the paired results, failed attempts, class-level errors, and resource measurements. Choose a recipe whose evidence supports your application, and reserve the final test for the frozen release decision.