Tune RAG retrieval with Polyaxon experiment matrices

A RAG configuration is a collection of interacting choices. Increasing retrieval depth may improve evidence coverage while adding irrelevant context; changing chunk size can alter both retrieval results and generation cost.

Use a Polyaxon experiment matrix to compare those choices under controlled conditions. Your RAG implementation supplies retrieval and generation; Polyaxon supplies repeatable execution, bounded parallelism, and a common place to compare the evidence.

Freeze the evaluation contract

Start with a versioned corpus and a fixed evaluation set. Each case should identify the question, expected evidence, and the criterion for an acceptable answer. Include questions that should produce an abstention.

Record the embedding configuration, prompt revision, evaluator revision, and provider settings. If these change between candidates, the experiment no longer isolates the parameters you intended to compare.

Separate the development set used for tuning from a held-out set used to confirm the selected configuration. Repeatedly selecting against one small dataset can produce a configuration that fits the benchmark better than the product.

Define a small matrix

The following operation compares two chunk sizes, three retrieval depths, and two model aliases: 12 candidates, with at most two child operations queued concurrently by the matrix.

It assumes your team has registered YOUR_ORG/rag-evaluation:v1 in the Component Hub. That component must declare the inputs shown and contain your RAG harness; it is not a bundled evaluator.

version: 1.1
kind: operation
name: rag-retrieval-sweep
hubRef: YOUR_ORG/rag-evaluation:v1

params:
  corpus_revision:
    value: corpus-v4
  dataset_revision:
    value: questions-v2
  evaluator_revision:
    value: grounded-answer-v3

cache:
  disable: true

matrix:
  kind: grid
  concurrency: 2
  params:
    chunk_size:
      kind: choice
      value: [256, 512]
    top_k:
      kind: choice
      value: [3, 5, 8]
    model_alias:
      kind: choice
      value: [candidate-a, candidate-b]

Replace the organization and aliases with your approved configuration. Define chunk_size and top_k as integer inputs and the remaining fields as strings. The harness must resolve each model alias to a recorded provider and model configuration.

The grid-search specification describes the search space and concurrency controls. Matrix concurrency does not limit how many provider requests each child process makes; bound that separately in the harness.

Keep index state candidate-specific

Changing chunk size changes the indexed representation. Do not let parallel candidates overwrite the same vector index or silently reuse an index built for a different configuration.

Give each index an identity derived from the corpus revision, chunking implementation, embedding configuration, and relevant parameters. Keep this identity with the run.

You can later reuse deterministic preparation through operation caching. First establish that the preparation is reproducible and that the cache identity includes every dependency that affects its output.

Attach configured connections only where needed. Keep provider keys out of parameters, run names, and artifacts.

Record evidence beyond one quality score

Have the harness report answer correctness, evidence coverage, abstention behavior, latency, and usage. Save per-case results so a high average cannot hide a regression on an important category.

Distinguish retrieval failure from answer-generation failure. A correct answer without the required evidence is different from a retrieved passage that the model misinterpreted.

Use tracking metrics for summaries and artifacts for approved detailed reports. Cost estimates should retain their pricing basis rather than appear as timeless provider facts.

Keep planned experiments separate from measured results. Defining a matrix does not establish that one configuration is already better.

Select and confirm the candidate

Compare candidates on the same completed cases using run comparisons. Keep failed and incomplete runs visible, even if they are excluded from the quality ranking.

Choose a configuration that satisfies the required quality floor, then examine its latency and resource tradeoffs. Confirm it on held-out cases before updating the application's defaults.

Persist the selected component, image, corpus, dataset, and evaluator revisions. The outcome is not simply “top-k equals five”; it is a reproducible configuration with evidence showing why it was selected.