Microservices on Kubernetes for ML platforms

Kubernetes makes it technically easy to deploy another service. That does not mean every model, pipeline step, or platform capability should become one.

A useful microservice boundary lets a team change, scale, secure, and recover a capability independently. A weak boundary adds a network hop and another deployment while leaving ownership and failure tightly coupled.

Start with capability and ownership

Define a service around a durable capability with a clear owner. In an ML platform, examples might include artifact metadata, experiment tracking, workload submission, feature retrieval, or online inference routing.

Ask these questions before splitting:

• Does the capability have a stable contract?
• Can one team own its runtime and on-call responsibility?
• Does it need to scale differently from its callers?
• Can it be deployed without coordinating every consumer?
• Can its data ownership be made explicit?
• Is the additional operational surface justified?

If the answers are mostly no, a module inside an existing service or an asynchronous job may be a better boundary.

Do not confuse workflow steps with services

Training, evaluation, data preparation, and batch inference are often finite workloads. Kubernetes Jobs and Polyaxon operations already give them execution boundaries, logs, status, resources, and retry behavior. Turning each step into a long-running HTTP service can add idle cost and a complicated control protocol.

Use a service when a capability must respond repeatedly to independent requests. Use a job when work has a defined start and completion. Use a scheduled operation when time triggers the work. The architecture should reflect lifecycle, not merely code organization.

Choose communication deliberately

Synchronous calls are easy to understand but couple availability and latency. A request that passes through several services inherits the failure and tail latency of the chain. For long ML tasks, returning a durable operation identifier and processing asynchronously is usually safer than holding an HTTP connection open.

Event or queue-based communication can absorb bursts and decouple timing, but it introduces delivery semantics, ordering, retries, and duplicate handling. Consumers must be idempotent where the transport can redeliver.

For every interaction, define:

Let Kubernetes provide primitives

Kubernetes contributes useful building blocks:

• Deployments and StatefulSets for lifecycle management;
• Services and DNS for discovery;
• readiness probes for traffic eligibility;
• ConfigMaps and Secrets for configuration inputs;
• service accounts and RBAC for workload identity;
• resource requests, limits, and autoscaling;
• NetworkPolicy for allowed traffic paths.

These primitives do not define the service contract, data model, or failure semantics. Kubernetes can restart a crashed container; it cannot decide whether replaying a model-registration request is safe.

Scale the bottleneck, not every component

Independent scaling is one of the strongest reasons for a service boundary. A lightweight API may need many replicas for request concurrency, while a GPU-backed model worker needs a smaller, carefully controlled pool. A queue can isolate those scaling policies.

Choose metrics that represent work. CPU utilization can be useful for a conventional API, but model services may need queue depth, request concurrency, time to first token, batch occupancy, or accelerator utilization. Scaling on the wrong signal can add replicas without increasing throughput.

The Horizontal Pod Autoscaler documentation explains the Kubernetes control loop. Capacity tests should still prove that the chosen metric leads to the desired latency and cost outcome.

Design for partial failure

Microservices convert some local failures into distributed ones. DNS, certificates, network policy, load balancers, queues, and downstream saturation become part of application behavior.

Keep failure handling explicit:

• use bounded timeouts;
• retry only transient and idempotent operations;
• apply backpressure instead of accepting unlimited work;
• isolate critical dependencies from optional ones;
• preserve operation state so long work can resume;
• expose readiness based on the ability to serve, not on every downstream dependency being perfect.

Correlate requests with Polyaxon runs when services submit or manage ML workloads. Platform observability should connect the API decision to the resulting operation, Pods, resources, and artifacts.

Prefer fewer, stronger boundaries

Begin with a cohesive service or modular application and extract only where evidence supports independence. Good signals include different scaling curves, separate security requirements, repeated deployment contention, or a capability used by several products.

Do not split solely to mirror an organization chart, copy another company's architecture, or make a diagram look cloud-native. Each new service adds a contract, deployment, identity, dashboard, alert policy, upgrade path, and failure mode.

For ML platforms, Kubernetes and Polyaxon already separate individual workloads from the services that orchestrate them. Use that distinction. Keep finite compute as operations, keep persistent capabilities as services, and add boundaries only when they make ownership or runtime behavior meaningfully clearer.