Build a conversational assistant on Kubernetes

A conversational assistant is more than a chat interface connected to a model. Production behavior comes from the complete system: identity, retrieval, prompts, model routing, conversation state, tools, policies, evaluation, and the services that keep the path available.

Kubernetes can operate these components, while an ML platform keeps their experiments, artifacts, and releases reproducible.

Define the assistant's contract

Write down who may use the assistant, which questions it should answer, which sources it may consult, which actions it may take, and when it must refuse or escalate. Define latency, availability, quality, freshness, privacy, and cost objectives.

Start with a bounded task and read-only behavior. Adding broad knowledge, many tools, and autonomous actions before the baseline works makes evaluation and incident response harder.

Identify the accepted outcome: a cited answer, correctly routed support case, extracted field, completed form draft, or another observable result. Avoid optimizing only for fluent responses.

Separate the application layers

A maintainable design separates:

• the user-facing API and session authentication;
• retrieval and source authorization;
• prompt and orchestration logic;
• model gateway or inference endpoint;
• tool execution and action authorization;
• conversation and task state;
• evaluation and observability.

Deploy services independently only when ownership, scaling, or failure behavior requires it. A small application can begin as one service plus managed dependencies while preserving these logical boundaries in code and telemetry.

Keep the request path traceable with stable conversation, turn, task, user or tenant, release, retrieval, and model-call identifiers. Do not place sensitive content in unrestricted metric labels.

Build grounded retrieval

Ingest approved documents through a versioned pipeline. Parse, normalize, chunk, enrich, embed, and index them while retaining the source URI, revision, access policy, and deletion state.

Apply user authorization before retrieval results enter the prompt. Hiding a source link after generation does not prevent the model from using restricted content. Keep tenant data and caches isolated.

Evaluate retrieval separately from generation. Measure whether the right passages were found, whether relevant sources were missed, and whether stale or unauthorized content appeared. Then measure whether the answer used the evidence correctly.

Control model access and routing

Use logical model aliases so the application does not embed credentials and provider details. A model gateway can apply authentication, token and concurrency budgets, routing, retries, fallbacks, and telemetry.

The Kubernetes Gateway API Inference Extension addresses model-aware routing for self-hosted inference. Adopt it or another gateway according to platform maturity, but keep the same operating requirements: visible routing decisions, bounded retries, and evaluation evidence for every eligible model.

Stream responses with cancellation and deadlines. Stop downstream generation when the client disconnects rather than consuming capacity for an answer nobody can receive.

Manage conversation state deliberately

Do not send an unlimited transcript on every turn. Retain the authoritative conversation according to privacy requirements, then construct context using a documented window, summary, or memory policy.

Version summaries and record which messages, retrieved passages, prompts, and tools formed each model input. Treat memory as application data with access, retention, correction, and deletion controls.

Keep coordination state outside the lifetime of a Pod. A replacement instance should resume or fail the turn explicitly instead of losing ownership or duplicating work.

Secure tools and data

Give the assistant a dedicated workload identity and only the network and data access required for its role. Keep model output separate from authorization. A generated tool call must pass schema validation, user and tenant authorization, policy, and action-specific confirmation where required.

Run untrusted code or file processing in an isolated workload with bounded resources, restricted egress, fresh storage, and no platform credentials. Scan uploaded and generated artifacts before promotion.

Redact secrets and sensitive content from logs. Define how operators gain temporary diagnostic access and how that access is audited.

Evaluate before and after release

Build a representative test set for task quality, grounding, citations, refusal, privacy, prompt injection, tool authorization, latency, and cost. Keep critical slices rather than only an aggregate score.

In production, sample appropriately protected outcomes, collect explicit feedback, and monitor retrieval misses, unsupported claims, policy stops, tool failures, model fallback, and cost per accepted task. Trigger reevaluation when the model, prompt, index, tool, policy, or user population changes.

Polyaxon pipelines can reproduce ingestion and evaluation, while tracked operations retain code, parameters, metrics, lineage, and artifacts. Deploy Kubernetes services with explicit resources and health checks, and use connections for approved models, indexes, and storage.

The first production milestone is not a human-sounding demo. It is a bounded assistant whose sources, behavior, authority, cost, and recovery path can be measured and explained.