Design search, retrieval, and recommendations with LLMs

Finding a document, selecting evidence for an answer, and recommending an item can share infrastructure while serving different goals. A useful design makes those goals explicit before choosing an embedding model or adding an LLM to the request path.

In Polyaxon, you can turn that design question into a bounded grid search: run the same query set against lexical, dense, and hybrid retrieval, with and without reranking. Keep the ranked results as artifacts and use the comparison dashboard to inspect the tradeoff instead of choosing from an architecture diagram alone.

Start with the result the user needs

Consider an engineering knowledge portal with three features: document search, answers to technical questions, and suggested training material.

Search can succeed by returning one exact reference page. A comparison answer may require several complementary passages. A recommendation may be useful even when it is deliberately different from the item the user just viewed.

Keep separate evaluation sets and acceptance criteria for these features, even if they initially query the same index.

Put hard constraints outside similarity

Represent access rights, product versions, language requirements, availability, and other mandatory conditions as explicit filters or eligibility checks.

An embedding match does not establish that a document is readable by the user or that a course is available in their region. An LLM may help interpret a request into a structured query, but the application must validate that query and apply trusted user and tenant context itself.

Preserve the original request alongside any rewrite. For an error-code search, a rewritten query that drops the code can remove the most useful signal. For a request with a negation, inspect whether the rewrite preserves what must be excluded.

Apply authorization before candidate content reaches an external reranker or generator. A final display filter is too late if restricted text has already crossed an unauthorized processing boundary.

Separate candidate generation from ranking

Candidate generation should find plausible items efficiently. Ranking spends more effort ordering a smaller set. The Sentence Transformers retrieve-and-rerank documentation describes this two-stage pattern using lexical or embedding retrieval followed by a cross-encoder that scores query-document pairs.

For the knowledge portal, compare lexical search, dense retrieval, and a hybrid candidate set. Keep exact identifiers available to lexical matching, and evaluate dense retrieval on paraphrased questions. Deduplicate candidates before measuring coverage or constructing answer context.

If a relevant document never enters the candidate set, a reranker cannot recover it. Examine candidate recall before attributing a poor final result to the ranking model.

Give the reranker a defined budget. Increasing candidate count adds scoring work and may affect the response deadline. Measure whether the additional candidates improve the user's result enough to justify that work.

Run six comparable retrieval candidates

The operation below enumerates three retrieval strategies and two reranking choices, with at most two candidate operations active concurrently. It assumes an application-specific image containing a retrieval_eval module, the polyaxon package, and a fixed evaluation configuration. That configuration must identify the corpus snapshot, indexes, relevance judgments, candidate count, and query set. The module implements the command-line flags shown here.

version: 1.1
kind: operation
name: compare-retrieval-strategies
matrix:
  kind: grid
  concurrency: 2
  params:
    retriever:
      kind: choice
      value: [lexical, dense, hybrid]
    reranker:
      kind: choice
      value: [none, cross_encoder]
component:
  inputs:
    - name: retriever
      type: str
    - name: reranker
      type: str
  run:
    kind: job
    container:
      image: registry.example.com/team/retrieval-eval:release-8
      command: [python3, -m, retrieval_eval]
      args:
        - "--config=/app/evaluation/docs-search-v3.json"
        - "--retriever={{ retriever }}"
        - "--reranker={{ reranker }}"
      resources:
        requests:
          cpu: "1"
          memory: 2Gi
        limits:
          cpu: "2"
          memory: 4Gi

The grid-search reference documents the parameter expansion and concurrency fields. Polyaxon supplies the candidate input values; your evaluation code selects the retrieval implementation and calculates the scores. Replace the example image and configure required index access with connections. For release qualification, pin the image and evaluation inputs to fixed revisions.

This worker configuration assumes any remotely served embedding and reranking models have their own resource allocation. If the runner loads a model locally, size that workload accordingly. Workflow concurrency limits jobs; each job's query parallelism and any shared endpoint quota need separate controls.

For each candidate, log candidate recall, ranking quality, retrieval duration, and failed-query count with tracking. Retain query IDs, ranked item IDs, relevance judgments, and timings in an artifact report. Keep answer-generation evaluation separate initially so a model change cannot conceal a retrieval regression.

The six combinations are six configurations, not six independent repetitions of each configuration. Repeat measurements when provider variation or stochastic steps are material, and do not count cached evaluation outputs as new observations.

Add generation where it has a clear role

An LLM can parse a natural-language request, propose a query expansion, summarize retrieved material, or explain a recommendation. These are separate experiments.

Keep the returned item IDs authoritative. A generated recommendation explanation must describe items selected by the retrieval and eligibility system; it should not invent a course or infer a qualification that the catalog does not contain.

For question answering, choose complementary evidence within the context budget and preserve citations. The knowledge-grounded application guide covers the decision to answer, clarify, or abstain when the evidence is insufficient.

For straightforward navigation queries, a ranked result list may already satisfy the task. Measure whether an added generation step improves completion rather than assuming every search needs a conversational answer.

Evaluate recommendations beyond similarity

For suggested training material, define usefulness in terms of the intended learning workflow. A highly similar item may repeat content the user already completed. A complementary prerequisite may be more valuable.

Include eligibility, novelty, catalog coverage, and user feedback alongside ranking quality. Inspect new users and new items separately because their available interaction history differs.

Clicks describe responses to what was shown. An item that received no exposure cannot be treated as rejected by every user. Retain impression and position information when interpreting interaction data, and use controlled online comparisons where appropriate to assess product outcomes.

Avoid training and evaluating on overlapping future behavior. Use data boundaries that reflect the information available when the recommendation would have been made.

Treat index updates as versioned work

Record document or item revisions, extraction and chunking settings, the embedding model, and index configuration. Keep the query encoder compatible with the representation used for stored vectors; matching dimensions alone does not establish semantic compatibility.

Use incremental updates for changed content when possible, and handle deletions explicitly. A removed document can otherwise remain discoverable through an old vector or cached result.

For an embedding-model migration, build and evaluate a compatible candidate index before changing the query path. Record a cutover plan and rollback conditions, including how permission and deletion updates remain effective during the transition.

Run the comparison workflow with Polyaxon

Once the retrieval comparison is useful, put corpus preparation and candidate index builds into a DAG, then pass their index identifiers into evaluation operations. The knowledge-grounded application guide includes an index-build operation with connection and resource configuration.

Record candidate coverage, ranking quality, end-to-end latency, and resource use separately. A single aggregate score makes it difficult to distinguish a better retriever from a slower reranker or a more selective access filter.

In the comparison dashboard, group runs by corpus and query-set revision before inspecting retriever and reranker choices. Use a scatter plot for the quality/latency tradeoff and open the case-level artifacts for exact-identifier and permission-sensitive queries. For recommendations, use a separate evaluation population with the appropriate exposure and time boundaries; a strong document-search result does not establish recommendation quality.

The application and search infrastructure own live authorization, index serving, and publication. Polyaxon provides repeatable execution and evidence around the experiments used to improve them.

Begin with one feature and a representative query set. Add complexity at the component whose measured failure prevents the user from completing the task.