Cache pipeline steps without hiding LLM regressions
Cache deterministic Polyaxon preparation steps while forcing fresh LLM evaluations, using explicit dependency identities and separate cache policies.
Caching can make an evaluation pipeline much cheaper when several candidates share the same data preparation. It can also hide a regression if yesterday's result is reused when the experiment was supposed to measure today's behavior.
Treat a cache hit as a claim that the relevant inputs and implementation are equivalent. Polyaxon's operation cache helps reuse work; your pipeline must make that equivalence explicit.
Separate the layers of caching
| Cache | What it reuses | Important distinction |
|---|---|---|
| Polyaxon operation cache | A previous operation's results | Depends on operation state and configured inputs |
| Container image cache | Downloaded image layers | Does not reuse the evaluation result |
| Provider prompt cache | Provider-specific request processing | Does not prove the response is identical |
| Application response cache | A saved answer | Must follow application freshness and access rules |
The operation-cache specification controls reuse at the workload level. Do not describe it as a semantic answer cache or a guarantee that an external model endpoint has not changed.
Cache preparation with an explicit identity
Consider a pipeline that normalizes documents, builds retrieval inputs, and evaluates several agent configurations. The normalization step can be a good reuse candidate if its output depends only on recorded inputs.
An operation might look like this:
version: 1.1
kind: operation
name: prepare-evaluation-corpus
hubRef: YOUR_ORG/prepare-corpus:v3
params:
corpus_digest:
value: REPLACE_WITH_CORPUS_CONTENT_DIGEST
normalization_revision:
value: normalize-v3
schema_revision:
value: documents-v2
cache:
disable: false
ttl: 86400This assumes a registered preparation component with matching string inputs. Replace the digest with one calculated from the actual source content, not a mutable filename or a label that is reused for different datasets.
The TTL bounds how long reuse is allowed; it does not discover unrecorded upstream changes. Include relevant code, tokenizer, embedding, and dependency revisions when they affect the prepared output.
Avoid excluding inputs from the cache identity until you can explain why changing them cannot alter the result.
Force fresh measurements where freshness matters
Keep the evaluation operation separate:
version: 1.1
kind: operation
name: evaluate-current-agent
hubRef: YOUR_ORG/agent-evaluation:v2
params:
dataset_revision:
value: regression-set-v5
candidate_revision:
value: agent-v12
evaluator_revision:
value: acceptance-v4
cache:
disable: trueThe registered component supplies your evaluation implementation and declares these inputs. Its input loading must reference the intended prepared data explicitly; ordering two operations does not automatically transfer workspace files.
Fresh execution is important when measuring provider behavior, stochastic outputs, latency, or operational reliability. Even a fully versioned configuration may need repeated trials because variability is itself part of the experiment.
Make reused outputs durable and attributable
Write preparation outputs through the configured artifact workflow. A temporary file left in a stopped container is not a durable input for another candidate.
Record the prepared artifact identity and producing run. Use lineage to connect consumers to the source of their data.
Review retention together with cache lifetime. If a policy removes the underlying artifact, the pipeline should not continue assuming it is available merely because the metadata still describes a reusable run.
Caching also does not authorize data access. Keep project permissions, storage permissions, and connections aligned with the consumer's intended access.
Review the cache policy as part of the experiment
For each step, document whether reuse is permitted and why. A practical review asks:
- Does every output-affecting dependency have a recorded identity?
- Are external mutable inputs excluded or explicitly versioned?
- Are repeated measurements supposed to be independent?
- Can downstream work still retrieve the reused artifacts?
- Will the report distinguish reused preparation from fresh evaluation?
Use run comparisons to compare outcomes under equivalent conditions. Do not count skipped work as a newly measured latency sample.
The useful pattern is selective reuse: cache stable preparation, rerun the behavior being evaluated, and preserve the connection between both. That reduces redundant compute without turning the evaluation into a replay of old conclusions.