Optimize LLM performance and cost with controlled experiments

An LLM application can spend less on each request and more on each completed task. A smaller model may require additional attempts. A shorter context may omit an exception that a reviewer must correct. A larger batch may improve throughput while making interactive users wait.

Optimization needs an experiment boundary that includes those consequences. The goal is an application configuration that meets its quality and service requirements at an acceptable total cost.

Use Polyaxon to keep the candidate configuration, evaluation results, usage quantities, and pricing assumptions in the same experiment record. Tracking, grid search, and the comparison dashboard make this a repeatable decision rather than a collection of disconnected token totals.

Freeze the question the experiment must answer

Consider a service that extracts structured fields from technical documents. Most documents are short, but some contain long tables and exceptions spread across several sections. The output feeds a reviewed internal catalog.

A useful first question is whether the service can process the same document population with less total cost while preserving field accuracy and its completion deadline.

Record the dataset revision, field-level rubric, review policy, model configuration, concurrency, cache state, and infrastructure. Include extraction failures, schema repairs, and human corrections. Compare the same document population rather than allowing an optimization to silently exclude difficult inputs.

Use LLM cost monitoring to establish the accounting boundary before changing the system. Preserve usage quantities and the pricing assumptions used for estimates, rather than relying on a currency total alone.

Log the estimate with its pricing basis

For a text-only evaluation, normalize provider usage into one record per attempt, including charged retries and failed attempts when usage is available. Each record should contain a pricing_key, uncached_input_tokens, cached_input_tokens, and output_tokens. The two input categories must be disjoint; do not count cached tokens again as uncached input.

The following script runs inside a Polyaxon job with the polyaxon package installed. It reads your normalized usage.json list and a reviewed pricing.json object containing a revision and a rates dictionary. Each pricing key maps the three token fields to their USD-per-million rates, recorded as decimal strings. Supply rates for your actual endpoint and contract; no provider prices are assumed here.

import json
from decimal import Decimal
from pathlib import Path

from polyaxon import tracking

usage = json.loads(Path("usage.json").read_text(encoding="utf-8"))
if not usage:
    raise ValueError("No usage records; zero cost is not established")
pricing = json.loads(Path("pricing.json").read_text(encoding="utf-8"))
fields = ("uncached_input_tokens", "cached_input_tokens", "output_tokens")
totals = {field: 0 for field in fields}
estimated_cost = Decimal("0")
for attempt in usage:
    rates = pricing["rates"][attempt["pricing_key"]]
    for field in fields:
        count = attempt[field]
        if type(count) is not int or count < 0:
            raise ValueError(f"Invalid token count: {field}")
        rate = Decimal(rates[field])
        if not rate.is_finite() or rate < 0:
            raise ValueError(f"Invalid price: {field}")
        totals[field] += count
        estimated_cost += Decimal(count) * rate / Decimal("1000000")

tracking.init()
tracking.log_outputs(pricing_revision=pricing["revision"])
tracking.log_metrics(
    **totals,
    recorded_attempts=len(usage),
    estimated_text_token_cost_usd=float(estimated_cost),
)
report_path = Path(tracking.get_outputs_path("cost-basis.json"))
report_path.write_text(
    json.dumps({"usage": usage, "pricing": pricing}, indent=2),
    encoding="utf-8",
)
tracking.log_file_ref(path=str(report_path), name="cost-basis")

The estimate covers only the supplied text-token buckets. Add separate accounting for other billed categories, tools, infrastructure, and reviewer effort. Do not treat missing usage after a timeout as zero; retain an unknown-cost flag and reconcile it when authoritative usage arrives. Unknown pricing keys intentionally fail instead of silently creating a cheap-looking result.

This uses the documented metrics and artifact APIs. Keep commercially sensitive rate cards in appropriately restricted artifact storage. Pair the cost result with the task acceptance report; a low token bill does not establish a useful completed task.

Locate the dominant cost and latency components

Separate document parsing, retrieval, model processing, output validation, retries, queue wait, and human review. Measure end-to-end duration as well as component timings.

For streamed responses, time to first token and time to a complete usable result answer different questions. For batch extraction, documents completed by the deadline may be more relevant than the speed of an individual model call.

Segment by document length, format, language, and task difficulty. The long-document subset may dominate cost while representing a small share of requests. An aggregate average can hide both that concentration and regressions affecting the subset.

Choose a small experiment matrix

Start with an intervention tied to the observed bottleneck:

Change one component first to establish a mechanism. Once its effect is understood, evaluate combinations. Two useful changes can interfere: context reduction may lower cache reuse, while routing may change the request distribution seen by the serving backend.

Keep the baseline in each comparison. Run candidates under comparable load and alternate their measurement windows when shared-service conditions might otherwise favor one candidate.

Use the retrieval matrix example as a pattern for a small parameter sweep. For the extraction application, start with two context budgets under one fixed model and rubric. Record cache state, request concurrency, and dataset revision as part of the candidate definition. Expand the search only after the first comparison identifies a useful direction.

Distinguish the kinds of caching

Caching an extracted document, caching a final answer, and reusing model prefix computation have different correctness and performance contracts.

An extraction cache needs a source revision and parser configuration. An answer cache also needs the question, relevant data revision, application configuration, and authorization scope. Similar wording alone is insufficient when users have different access or the underlying facts have changed.

For self-hosted inference, vLLM's prefix-caching documentation explains that reuse reduces processing of shared input prefixes; it does not accelerate generation of new output tokens. Measure the input-processing and output-generation portions before predicting its benefit.

Report cold and warm behavior. A benchmark that repeatedly submits the same documents can overstate cache benefits for a production stream dominated by new material. Also record expiry, invalidation, and storage costs.

Evaluate smaller models and routing end to end

Compare candidates on the exact field definitions and input population. Include malformed output, missed exceptions, and correction effort. A model that is adequate for a bounded extraction task may still fail on documents requiring cross-section interpretation.

If routing is used, record which documents take each path and which trigger fallback. Include the first attempt's cost when a stronger model handles the retry. Evaluate the router and the selected model as one application configuration.

Keep deterministic eligibility rules outside the model. A document type that requires an approved processing environment cannot be sent to another provider merely because the router predicts lower cost there. See evaluating LLM routers for route-level measurement.

Include the operating cost of self-hosting

Compare hardware allocation, idle capacity, redundancy, storage, network, and engineering support against provider usage. Include the capacity needed for peak demand and the quality requirements of the actual model being served.

Quantization and serving changes require fresh quality and load measurements. A smaller memory footprint does not guarantee better latency for every hardware and workload combination. Keep model format, serving configuration, hardware, and workload distribution in the experiment record.

Avoid a universal traffic threshold for moving to self-hosting. The decision depends on utilization, model suitability, reliability requirements, and the team's ability to operate the service.

Use resource monitoring for workload CPU, memory, and GPU measurements when enabled, and retain the actual resource allocation and measurement window. Apply shared scheduling presets so comparison jobs use the intended environment. In Polyaxon EE and Cloud, queues can separate offline experiments by concurrency, priority, quota, and execution destination. Endpoint request throttling remains an application or serving-layer concern.

Keep optimization experiments connected in Polyaxon

Use a bounded Polyaxon grid search to enumerate candidate configurations. DAG operations can prepare shared fixtures, execute candidates, and aggregate results.

Log measurements and configuration with tracking, and preserve case-level outputs and error reviews as artifacts. Control workload concurrency alongside application-level provider limits.

In the comparison dashboard, first filter to the same dataset and rubric. Put estimated cost on one chart axis and acceptance rate on the other; inspect latency and failure count before selecting a candidate. Open the cost-basis artifact if a result looks unusually cheap, especially after changing a model route or usage-normalization code.

Qualify candidates against minimum quality and service requirements before comparing their cost. Retain uncertainty and sample size with the result; a small apparent saving may be ordinary variation.

Roll out the selected configuration gradually and compare its actual workload mix with the evaluation population. A successful optimization should remain explainable: what work was removed or accelerated, which requirements still hold, and whether the expected savings survived real traffic.