Model development
Develop a model from a reproducible baseline to a reviewed candidate, with versioned data, comparable evaluations, and a clear handoff to deployment.
Model development is a sequence of decisions supported by experiments. A useful result includes the trained model, the conditions that produced it, the evidence for choosing it, and the information another engineer needs to use it.
Consider a team building a classifier that routes support tickets. Improving overall accuracy is useful only if the model also handles important ticket categories, respects the response-time budget, and can be reproduced when the training data changes.
Define the decision before training
Start with a baseline and an acceptance contract. The baseline might be an existing model or a simple rules-based classifier. Evaluate it on the same cases as each candidate so a new approach has something concrete to improve.
| Decision | Evidence to retain |
|---|---|
| Does the model route tickets correctly? | Per-category precision and recall, confusion matrix, and error examples |
| Does it help the important cohorts? | Results by language, ticket source, and category |
| Can the service operate it? | Prediction latency, model size, and required runtime |
| Is the candidate worth promoting? | Comparison with the baseline and the reason for accepting the tradeoff |
Choose thresholds with the people who own the workflow. Missing an urgent ticket and incorrectly escalating an ordinary ticket have different consequences; one aggregate score cannot express both.
Freeze the data and evaluation boundary
Keep an identifiable training snapshot, validation split, and final test split. If tickets from the same conversation or customer are related, account for those groups when splitting; for a time-dependent task, preserve the intended time boundary.
Fit preprocessing on training data. Use validation results to choose features, parameters, and stopping rules, then assess the selected approach on the untouched test set. Repeatedly choosing models from test results turns that set into another tuning input. The scikit-learn guidance on data leakage explains why preprocessing and splitting must respect this boundary.
Record the dataset manifest, split definition, label taxonomy, and transformation revision. A path such as latest/tickets.csv does not identify the bytes used in an earlier experiment. Use a retained snapshot or immutable object version, and store a digest when available.
Make each experiment explain itself
An experiment should answer a specific question, such as whether class weighting improves recall for urgent tickets without an unacceptable increase in false escalations. Keep unrelated settings fixed while investigating that question.
Use Polyaxon tracking to record inputs, measured metrics, artifacts, and the execution context. The experiment-tracking walkthrough shows how to save a model file and supporting reports. Record at least:
- Code revision, dependency environment, and container image where applicable.
- Dataset and split identities, random seed, and training parameters.
- Validation metrics, cohort results, and representative errors.
- Model artifact and the preprocessing assets needed to load it.
Move repeatable training into a component and operation when local iteration is ready for shared compute. The training program still owns fitting and evaluation; Polyaxon supplies the execution and tracking context around it.
Compare candidates and inspect the tradeoff
Use the run comparison dashboard to compare candidates evaluated under the same contract. A higher score on a different validation snapshot is not evidence of improvement over the baseline.
Inspect failures before expanding the search. If nearly every urgent-ticket error comes from a mislabeled source, a larger parameter sweep may optimize the wrong signal. For promising changes, repeat runs where randomness materially affects the decision and retain the individual results alongside their summary.
Promote an identifiable model package
Saving a checkpoint is different from selecting a model version. Once the evidence supports a candidate, use model registration or promotion to associate a named version with its producing run and chosen artifacts.
The package should include preprocessing, feature order, label mapping, and runtime requirements as well as model weights. Retain the evaluation report and decision record so the next person can understand why this candidate was selected. A registry version identifies that package; production approval and deployment are separate decisions.
Close the loop after deployment
Deployment creates new evidence: response times, errors, changed input distributions, and eventual task outcomes. Connect those observations to the deployed model version and use them to decide whether to investigate data quality, adjust the application, or begin a new development cycle.
The handoff is complete when another engineer can retrieve the selected package, reproduce its evaluation under the recorded conditions, and identify the baseline to restore if the change performs poorly. That makes model development reusable engineering work rather than a collection of promising notebook outputs.