Build an LLM pipeline to classify ML failure reports

A platform team has a recurring review task: read failed ML workload reports and route them to the right owner. Some failures concern input data, others resource limits or missing dependencies. An LLM classifier can propose a category, but the useful deliverable is a reviewable result connected to the exact evidence it classified.

Polyaxon provides the surrounding building blocks: run metadata queries, artifact retrieval, DAG dependencies, and experiment comparison. This example uses those capabilities to develop a classifier. The classification code and taxonomy belong to your application; Polyaxon does not infer the failure category for you.

Define the routing decision

Start with a small taxonomy that a platform engineer can apply consistently:

These categories are illustrative. Adapt them to the failures your team actually sees. A Python exception in a log is not enough to establish its root cause, and an unavailable report should remain an ingestion failure rather than receive a guessed label.

Keep the first version advisory: propose a route and supporting evidence, then let an owner confirm it. Do not let the classifier restart jobs, change resources, or execute remediation commands found in a log.

Assemble an authorized, versioned dataset

Query the projects your workflow is permitted to inspect. Polyaxon's RunClient query interface can list runs and retrieve their statuses, inputs, outputs, artifact lineage, and selected artifacts. Follow pagination when collecting a dataset; one list response is not necessarily the complete run population.

Create one sanitized record per failure incident. Include a source run ID, workload revision, report digest, normalized status information, and only the log excerpts required for classification. Remove credentials, customer payloads, and unnecessary personal information before passing the record to a model provider.

Save a manifest describing the selected runs, extraction code revision, taxonomy revision, and split assignment. Related retries, restarts, and nearly identical reports belong in the same split. Otherwise, a held-out evaluation may reward recognition of an incident already seen during prompt development.

Build a reviewed qualification set separately from any model-generated training labels. The final comparison should measure agreement with resolved human judgments, not just agreement with a teacher model.

Require evidence with every proposal

The following record illustrates the output contract, not a result from a real customer workload:

{
  "incident_id": "fixture-042",
  "source_run_id": "source-run-reference",
  "report_revision": "sanitized-reports-v4",
  "taxonomy_revision": "failure-routing-v2",
  "proposed_category": "needs_review",
  "evidence_ids": ["status-1", "excerpt-3"],
  "reason_code": "termination_cause_unconfirmed",
  "review_status": "pending"
}

Validate the category against an allowlist and check that every evidence ID exists in the supplied report. Keep the proposal separate from the reviewer's final category. Model confidence, when supplied, is another output to evaluate; it is not automatically a calibrated probability.

Treat logs and exception messages as untrusted input. A command inside a report is evidence to classify, not an instruction for the pipeline to execute.

Connect preparation, classification, and scoring

Create three reusable components: one assembles the sanitized dataset, one runs a classifier, and one scores the proposals against reviewed labels. This template assumes you have registered those application-specific components under the illustrative names shown below.

The preparation component declares a string output named manifest_uri. Classification accepts that URI and declares a string output named predictions_uri. Scoring accepts both. The producer programs record these outputs with tracking.log_outputs; the consumers read the referenced files using their configured store access.

version: 1.1
kind: component
name: failure-report-classification
run:
  kind: dag
  operations:
    - name: prepare
      hubRef: your-team/prepare-failure-reports:1.0
      params:
        dataset_revision: {value: sanitized-reports-v4}
    - name: classify
      hubRef: your-team/classify-failure-reports:1.0
      params:
        manifest_uri:
          ref: ops.prepare
          value: outputs.manifest_uri
        classifier_revision: {value: failure-classifier-v2}
    - name: score
      hubRef: your-team/score-failure-reports:1.0
      params:
        manifest_uri:
          ref: ops.prepare
          value: outputs.manifest_uri
        predictions_uri:
          ref: ops.classify
          value: outputs.predictions_uri

The parameter references establish the upstream dependencies, as described in the DAG dependency reference. Passing a URI does not copy an upstream container's filesystem into the next job. Write results to an accessible durable store, use a distinct output location for each attempt, and supply the necessary connections in each component.

The preparation component needs access to selected source reports. The classifier should only need the sanitized dataset. The scoring component needs reviewed labels and predictions, but does not need the model provider credential. Keep those access requirements separate.

Compare classifiers in the runs dashboard

Begin with a rule-based baseline for explicit failure signals, then compare a prompted model on the same qualification manifest. Add fine-tuning only when the labeled dataset and recurring failure patterns justify that experiment.

Have the scoring component log total incidents, missing predictions, per-category precision and recall, review rate, and incorrect automatic routes through tracking. Define each denominator in the report. A category absent from the evaluation set has no measured recall; do not manufacture a zero or a perfect score for it.

Use the comparison dashboard to select runs with the same dataset and taxonomy revisions. Compare the proposed automation rate alongside the error rate among automatically routed incidents. A classifier that asks a person to review almost everything can look accurate without saving much triage work.

Save the confusion table, per-incident predictions, and reviewer disagreements as artifacts. Retain unresolved and failed classifications as well as successful ones. The lineage view connects those reports to their producing operations and workflow dependencies.

Make corrections reusable

Store reviewer corrections as a new labeled dataset revision instead of overwriting the original proposal. Record whether an update changes the report extraction, taxonomy, classifier, or qualification split. Reevaluate the baseline when the measurement rules change.

Only a separate, authorized publishing step should update an incident's official routing record. Make that write idempotent using the incident and decision revision. A retry of the classifier should not create duplicate tickets or repeatedly notify an owner.

The first useful milestone is modest: a reviewer can open a proposal, inspect the evidence, correct it, and find the experiment that produced it. The next milestone is a measured reduction in triage work without unacceptable routing errors. Keep both the selected and rejected approaches discoverable using the experiment evidence workflow.