Evaluate zero-shot prompting for production tasks

Suppose your ML platform receives training failures that need routing to a data, infrastructure, or model owner. A zero-shot prompt can classify each report using instructions and a defined output format, without examples in the request. The practical question is whether that simple configuration handles your actual failure reports reliably enough to automate routing.

Polyaxon makes this a repeatable experiment. Store the classification contract and prompt in versioned code, run candidates as parameterized jobs, retain each case result, and compare accuracy with latency and token usage. The same platform can then schedule the chosen workflow and rerun its evaluation when the prompt, model, or input population changes.

Define the classifier's contract

Choose explicit labels, such as data, infrastructure, model, and needs_review. Specify the evidence required to route a report and when an ambiguous or incomplete report should receive needs_review. Treat JSON validity and label correctness as separate checks.

Build a fixed case manifest containing a stable case ID, sanitized report, expected category, and any relevant slice such as training framework or failure type. Include examples with misleading wording, multiple possible causes, and missing evidence. Keep expected labels out of the model input.

Use separate development and qualification manifests. Prompt edits can use development failures, but the final selection should be measured against held-out cases. Log the manifest digest so two runs described as using the same dataset can be compared precisely.

Package prompt variants as one component

Put the inference harness, parser, and scoring code in one Polyaxon component. Declare inputs for prompt strategy, prompt revision, model, inference settings, and case-manifest revision. Build the required provider client and dependencies into the component's image.

Start with a zero-shot baseline and a few-shot variant using reviewed examples. Hold the model, input cases, output contract, and evaluator constant. If retrieval is necessary to obtain the facts behind a failure, evaluate it as an explicit additional strategy.

The model call remains application code. Polyaxon's LLM integration examples illustrate provider and tool integration patterns; job evaluation does not require executing prompts through a sandbox. Attach only the connections the harness needs to access its provider and case data.

Record comparable evidence in the job

Use the following helper inside a managed Polyaxon evaluation job whose image includes the Polyaxon client and whose tracking and artifact collection are enabled. Your inference harness calls it once after collecting results.

The case manifest is a JSON array with case_id, report, and expected fields. Each outcome supplies case_id, parsed category (or null), a Boolean schema_valid, and measured latency_ms, input_tokens, and output_tokens. These values come from the actual application and provider response. This helper records and scores them; it does not call a model.

import hashlib
import json
from pathlib import Path

from polyaxon import tracking
from polyaxon.schemas import V1ArtifactKind


def record_evaluation(case_path, outcomes, strategy, prompt_revision, model):
    case_bytes = Path(case_path).read_bytes()
    cases = json.loads(case_bytes)
    expected = {case["case_id"]: case["expected"] for case in cases}
    actual = {row["case_id"]: row for row in outcomes}
    if not cases or len(expected) != len(cases):
        raise ValueError("Expected nonempty cases with unique case IDs")
    if len(actual) != len(outcomes) or set(actual) != set(expected):
        raise ValueError("Outcomes must cover every case exactly once")

    tracking.init()
    tracking.log_inputs(
        strategy=strategy,
        prompt_revision=prompt_revision,
        model=model,
        case_manifest_sha256=hashlib.sha256(case_bytes).hexdigest(),
    )

    report = []
    for case_id, label in expected.items():
        row = actual[case_id]
        report.append({
            **row,
            "expected": label,
            "correct": row["schema_valid"] is True and row["category"] == label,
        })

    count = len(report)
    tracking.log_metrics(
        accuracy=sum(row["correct"] for row in report) / count,
        schema_valid_rate=sum(row["schema_valid"] is True for row in report) / count,
        mean_latency_ms=sum(row["latency_ms"] for row in report) / count,
        input_tokens=sum(row["input_tokens"] for row in report),
        output_tokens=sum(row["output_tokens"] for row in report),
    )
    path = tracking.get_outputs_path("evaluation/cases.json")
    Path(path).write_text(json.dumps(report, indent=2), encoding="utf-8")
    tracking.log_artifact_ref(
        path=path,
        kind=V1ArtifactKind.FILE,
        name="classification-cases",
    )

Invalid output counts as a failure instead of silently disappearing from the denominator. The complete per-case report supports later slice analysis. Keep the parser strict and validate outcome field types and ranges in the harness before calling the helper.

The file is written to the run's outputs directory before its artifact reference is registered. The report's existence means evaluation evidence was retained; it does not imply the candidate met its release threshold.

Sweep strategies and inspect the results

Use a grid-search matrix over the component's declared strategy input to launch the baseline and variants. Set matrix concurrency according to provider quotas and the workload queue. Record retry settings because rate-limit retries can affect observed latency and spend.

Open run comparison and filter to the same manifest digest and model configuration. Display strategy, prompt revision, accuracy, schema validity, latency, and token counts. Compare the full case artifacts for candidates whose averages look similar.

A few-shot prompt may reduce misrouting while adding input tokens. A zero-shot candidate may be faster but mishandle a rare infrastructure failure with a high operational cost. Inspect false routing and unnecessary review separately before selecting a default.

Turn the selected prompt into an operated workflow

Record the chosen component and prompt revisions, qualification run, acceptance criteria, and known weak cases. If you calculate cost, log the provider usage and pricing basis used for that calculation; total tokens alone are not a currency amount.

Use the same component for subsequent qualification runs. Scheduled operations can rerun a fixed regression set, while a newly reviewed case manifest captures changes in real failure patterns. This keeps the production prompt connected to its evaluation history and the wider Polyaxon workflow that depends on its decisions.