Production LLM systems: Where to invest after the prototype

Your first LLM prototype works. The next three requests are less clear: improve the answers, reduce latency, and make releases safer. Before turning those requests into separate infrastructure projects, give the team one repeatable way to compare the current application with a proposed change.

In Polyaxon, that starts with an evaluation operation that records its inputs, task-level results, and aggregate metrics. The runs comparison dashboard then becomes a place to investigate an engineering decision: which change helped, on which cases, and at what cost?

This guide uses a proposed supplier-comparison assistant as an example. The workflow and measurements below are a starting design, not customer results or a claim about typical production performance.

Start with the work that reaches a user

Consider an internal procurement assistant that prepares supplier comparisons. Its output must use eligible suppliers, reflect the requested delivery region, and identify information that needs verification. Procurement staff make the final decision.

For this application, a completed generation is only an intermediate event. The useful outcome is a comparison the reviewer can accept with a known amount of correction. Measure that outcome alongside turnaround time and total operating cost.

Define the unit of work before choosing a dashboard. One supplier comparison may require several searches, model requests, corrections, and human interactions. Count the complete task once, and include failed attempts in its cost. Record abandoned tasks separately so a growing backlog cannot disappear from the success calculation.

Use a fixed review rubric initially. If reviewers change their interpretation of an acceptable comparison, version the rubric and reassess a shared sample. Otherwise, a quality trend may reflect changing judgment rather than changing application behavior.

Give every candidate the same result contract

Have your evaluation program produce a JSON document containing dataset_revision, rubric_revision, candidate_revision, pricing_basis, and a tasks array. Each task record should contain a unique task_id, a reviewed boolean accepted, end-to-end duration_ms, and estimated_cost_usd covering all its attempts. Include failed and timed-out tasks in the array. Reconcile it against the expected task manifest before aggregation; an empty or incomplete result file is not a successful evaluation.

The following aggregation code runs inside a Polyaxon job with the polyaxon package installed. It reads your evaluator's output; it does not generate judgments or retrieve provider prices. Cost estimates must use an explicitly recorded pricing basis.

import argparse
import json
from pathlib import Path

from polyaxon import tracking

parser = argparse.ArgumentParser()
parser.add_argument("--results", required=True)
args = parser.parse_args()
report = json.loads(Path(args.results).read_text(encoding="utf-8"))
tasks = report["tasks"]
if not tasks:
    raise ValueError("An evaluation must include tasks")
if len({task["task_id"] for task in tasks}) != len(tasks):
    raise ValueError("Duplicate task IDs would distort the comparison")
if any(type(task["accepted"]) is not bool for task in tasks):
    raise ValueError("accepted must be a reviewed boolean")

accepted = sum(task["accepted"] for task in tasks)
total_cost = sum(task["estimated_cost_usd"] for task in tasks)
metrics = {
    "tasks_total": len(tasks),
    "accepted_tasks": accepted,
    "acceptance_rate": accepted / len(tasks),
    "mean_duration_ms": sum(task["duration_ms"] for task in tasks) / len(tasks),
    "estimated_cost_usd": total_cost,
}
if accepted:
    metrics["estimated_cost_per_accepted_task_usd"] = total_cost / accepted

tracking.init()
tracking.log_outputs(
    dataset_revision=report["dataset_revision"],
    rubric_revision=report["rubric_revision"],
    candidate_revision=report["candidate_revision"],
    pricing_basis=report["pricing_basis"],
)
tracking.log_metrics(**metrics)
saved_report = Path(tracking.get_outputs_path("evaluation-report.json"))
saved_report.write_text(json.dumps(report, indent=2), encoding="utf-8")
tracking.log_file_ref(path=str(saved_report), name="evaluation-report")

This uses documented metadata logging and artifact logging. Fixed revision identifiers go into outputs, numeric measurements into metrics, and the detailed report into artifact storage. When no task is accepted, cost per accepted task is undefined; do not display it as zero or allow that candidate to win a cost comparison.

Use sanitized task records in the report. The application should validate numeric values, expected case coverage, and reviewer provenance before this aggregation step.

Build a loss map for one workflow

Follow a small, representative set of tasks from request to accepted result. At each point, identify the work lost and the component that can change it.

Treat each row as a hypothesis. An apparently slow model might actually be waiting behind another workload. A retrieval change might improve the average result while harming a smaller but important language segment. Inspect examples before committing to a large platform change.

Prioritize by consequence and leverage

Write a short investment note for each candidate: affected tasks, observed failure frequency, consequence, owner, proposed change, and evidence that would demonstrate improvement.

Some controls are release prerequisites. An application that exposes restricted supplier contracts needs access enforcement before broader deployment, even if those requests are uncommon. Other improvements compete for engineering time. Reducing a frequent manual correction may be more valuable than making an already acceptable answer slightly faster.

Shared fixes can benefit several applications, but verify that their requirements actually match. A common evaluation runner is useful across teams; a single quality rubric for procurement, support, and code generation is unlikely to be sufficient.

Keep the backlog small enough to evaluate its effects. If the team changes the model, retrieval strategy, prompt, and routing policy together, a better result will be difficult to explain and a regression difficult to isolate.

Include the operator in the experiment

Record the work required after something goes wrong. Can an operator identify the affected application version? Can the reviewer recover a saved comparison? Does disabling an unreliable supplier connector stop new requests without losing already accepted work?

A useful improvement might reduce investigation time or make an escalation actionable without changing model accuracy. Include those outcomes in the acceptance criteria.

Exercise the specific failure that motivates the investment in a controlled environment. For a connector fix, interrupt a response and inspect the resulting task status. For a release-process fix, deliberately introduce a known bad comparison and confirm that the qualification process identifies it. The agent deployment checklist provides a more detailed launch rehearsal.

Inspect the run before buying more capacity

Select the baseline and candidate in the comparison dashboard. Display their dataset and rubric revisions alongside acceptance rate, duration, and estimated cost. First confirm that the compared runs measured the same population. Then open the reports to inspect changed outcomes; an average alone cannot tell you whether an important supplier category regressed.

If the problem is elapsed time, inspect the run's statuses and resource monitoring. Queue delay, startup delay, and execution time need different remedies. CPU, memory, and GPU monitoring, when enabled, can help identify an under-resourced preprocessing step. Provider waiting time must be measured by your application; low CPU usage does not establish spare endpoint capacity.

For teams using Polyaxon EE or Cloud, queues add workload concurrency, priority, quota, and cluster-routing controls. Those govern scheduled operations, not the number of model requests each worker creates.

For recurring qualification, express preparation, candidate execution, scoring, and report generation as dependent operations in a DAG. Your scoring code defines acceptance; the workflow makes those steps repeatable and connects their outputs.

Keep a record of the baseline, intervention, evaluation population, and decision. If the change affects a provider-managed model or live data source, note which aspects could not be frozen. Reproducible configuration supports comparison even when identical generated text cannot be guaranteed.

Review investments against their original promise

After rollout, revisit the task population and conditions used to justify the change. Check whether accepted work increased, correction effort fell, and important segments remained healthy. Include deployment and maintenance effort in the assessment.

Retain rejected ideas as well as successful ones. Knowing that a larger context window failed to improve supplier comparisons can save another team from repeating the same experiment.

The next production investment should have a clear connection to an observed problem and a measurable outcome. Keep the run IDs, reviewed report, and rollout decision together so the next engineer can inspect the evidence. The LLMOps maturity assessment helps turn those records into a capability roadmap.