A fuller picture of model behavior during training

A single metric rarely tells the truth about a model. Accuracy can improve while a critical slice gets worse. Loss can move in the right direction while generated samples degrade. A model can look fine in aggregate and fail on the examples the business actually cares about.

That is why experiment tracking needs more than scalar metrics.

For example, consider these illustrative counts from the same 1,000-ticket validation set:

The new candidate improves the aggregate while making 20 additional errors on urgent tickets. These are explanatory numbers, not benchmark results. A cohort report and the underlying failed examples make that tradeoff visible before the team selects a model.

Track the data profile

A training run should make it easy to inspect the data it used. Dataset versions, schema changes, missing values, class balance, summary statistics, and sample rows all matter. Without that context, a model regression turns into archaeology.

For tabular and structured data, summary statistics help catch obvious problems early: shifted distributions, new null patterns, broken joins, and unexpected category growth. For image and text workloads, logged samples help teams inspect whether preprocessing did what it was supposed to do.

Log artifacts, not just metrics

Useful model debugging often depends on artifacts:

• Prediction samples.
• Confusion matrices.
• ROC and precision-recall curves.
• Generated images.
• Feature importance reports.
• Evaluation notebooks.
• Model files and checkpoints.

Those artifacts need to be tied to the run that produced them. Saving them somewhere in object storage is not enough if nobody can answer which code, data, parameters, and environment produced the artifact.

Keep a stable case identifier in a prediction report, along with the split, cohort, expected label, and predicted label. Then attach the report and its summary to the same run as the aggregate metrics. After your evaluation program has written predictions.csv and slice-report.json, a Polyaxon job with the SDK and tracking configured can save them with:

from polyaxon import tracking
from polyaxon.schemas import V1ArtifactKind

tracking.init()
tracking.log_artifact(
    path="predictions.csv", name="prediction-cases", kind=V1ArtifactKind.CSV
)
tracking.log_artifact(
    path="slice-report.json", name="slice-report", kind=V1ArtifactKind.FILE
)
tracking.end()

The evaluator computes those files; the logging calls retain them and their lineage. Keep only the examples and fields the team is permitted to inspect. The artifact logging guide also covers images, curves, and reference-only logging when the content lives in a separate store.

Inspect behavior over time

For computer vision and generative workloads, logging images during training can reveal behavior that metrics hide. You can see whether predictions stabilize, whether augmentations are corrupting inputs, and whether a model is learning meaningful structure or memorizing noise.

The same idea applies outside images. Text generations, ranked examples, failed predictions, and slice reports give teams concrete evidence instead of another dashboard full of averages.

Use a fixed sample panel when comparing checkpoints so changes in the displayed examples do not masquerade as changes in model behavior. Record the sample selection and preprocessing revision, and keep each checkpoint's artifact associated with its training step. A separate randomly selected panel can help reveal whether the fixed cases are unusually easy.

How Polyaxon fits

Polyaxon tracks metrics, logs, artifacts, lineage, and runtime metadata for each run. That gives teams one place to compare experiments, inspect outputs, understand data and model versions, and move from a failed metric to the files and logs that explain it.

Start from run comparison, confirm that the candidates share the intended dataset and evaluation contract, then inspect their cohort reports and prediction artifacts. For the ticket example, the next decision is whether the ordinary-ticket improvement justifies the urgent-ticket regression, not simply which row has the highest accuracy.