Design reliable self-hosted ML infrastructure

A healthy API does not prove that ML work is progressing. Users can submit runs while a queue is stalled, Pods can start while artifact storage is unavailable, and a training process can continue after its checkpoints stop reaching durable storage.

Reliability for a self-hosted ML platform is the ability to accept, execute, observe, and recover work without silently losing its state or evidence. That requires more than redundant servers. Each part of the path needs a failure model, an owner, and a tested recovery action.

Define the service from the user's workflow

Start with the promises users depend on. A practical service path may include:

1. Authenticate and validate a submission.
2. Commit the operation and its inputs.
3. Admit and route it to an eligible execution environment.
4. Create the Kubernetes workload.
5. Start the application with its data and credentials.
6. Stream status, logs, and metrics.
7. Preserve checkpoints, artifacts, and terminal state.
8. Make the result discoverable and retrievable.

Define indicators at these boundaries. API availability, submission durability, queue age, scheduling delay, startup success, active progress, artifact-write success, and terminal-state consistency explain different failures.

Avoid a single “platform uptime” percentage that treats every green process as equivalent. The AI SLO guide covers user-centered targets and error budgets; this article focuses on the infrastructure design that supports them.

Draw failure domains before adding redundancy

Inventory the components and the scope of each failure. Include the load balancer, identity provider, API, database, message or operation queue, Kubernetes control plane, worker nodes, container registry, artifact store, DNS, certificate path, telemetry backend, and external services.

For each dependency, record:

Redundancy within one shared dependency may not create an independent failure domain. Replicas can share a database, network, region, identity provider, configuration error, or exhausted quota. Describe the common causes that can affect them together.

Separate the control plane from workload execution

The platform control plane stores desired state, policy, identity mappings, queue position, and execution metadata. Compute environments run user workloads. Isolating their resources and permissions reduces the chance that a noisy or hostile job degrades the API or corrupts platform state.

Use separate namespaces, node pools, service accounts, quotas, and network policy at minimum. Harder isolation requirements may justify separate clusters or accounts. Keep the control-plane database and artifact metadata outside disposable worker filesystems.

Self-hosted Kubernetes also needs its own availability design. The Kubernetes high-availability kubeadm guide documents stacked and external-etcd control-plane topologies. Managed Kubernetes changes who operates these components, not the need to understand their regional, quota, identity, and upgrade failure modes.

The goal is not that every control-plane outage instantly terminates running containers. It is that operators know which functions continue, which state is safe, and how the platform reconciles reality when control returns.

Classify state by recovery importance

Not every byte needs the same availability or backup policy.

Do not make task recovery depend on sampled telemetry. A trace can explain a failure, but the committed checkpoint and operation record should independently determine what runs next.

Backups need a consistency story across systems. Restoring a database to one point and artifacts to another can leave records referring to missing files or untracked outputs. Define restore ordering, reconciliation, and the evidence that proves the restored boundary is usable.

Preserve accepted work through interruptions

Commit a submission before acknowledging it. Give the logical operation a stable identity independent of a particular Pod or node. A retry by the client should not create duplicate work unless the user explicitly requests another run.

Queues must survive the failures they are expected to bridge. Track accepted, eligible, dispatched, running, retrying, blocked, canceled, and terminal states explicitly. Give leases and retries time bounds, and prevent a stale worker from reporting success after ownership moved elsewhere.

For long workloads, save checkpoints to storage available from the recovery destination. Validate the complete resume path with a real interruption. A file written to local disk or a checkpoint that restores weights but not optimizer, data, or application state may not meet the recovery objective.

Keep external side effects separate from compute retries. A repeated notification is inconvenient; a repeated data mutation or deployment can be damaging. Record action identities and receipts, then reconcile unknown outcomes before another attempt.

Make failover eligibility explicit

Failover is a routing decision constrained by workload compatibility. A second cluster or cloud is not a fallback if it lacks the dataset, accelerator, image, credentials, controller, network path, artifact store, or permitted region.

Define hard eligibility first. Rank eligible destinations by capacity, expected queue delay, transfer time, cost, and current health. Record the reason for every reroute so operators can distinguish planned failover from configuration drift.

Decide whether failover applies to new submissions, queued work, interrupted work with checkpoints, or live services. Each case has different consistency and recovery requirements. Automatically starting a second attempt while the first may still be running can violate singleton or side-effect assumptions.

Use circuit breakers and backoff to keep a failing destination from attracting repeated work. Retain a manual override and a fail-closed path for cases where automated routing cannot preserve data, security, or correctness requirements.

Use telemetry as part of the control loop

Metrics summarize state over time, logs preserve discrete records, traces connect work across services, and Kubernetes events describe orchestration decisions. OpenTelemetry documents these observability signals as complementary sources rather than substitutes for one another.

Connect platform signals with stable operation, attempt, project, queue, agent, cluster, namespace, and workload identities. An operator should be able to move from an aged queue to the rejected scheduling constraint, from a failed artifact upload to the storage path, and from a user-visible delay to the responsible dependency.

Alert on service impact and exhausted recovery margin:

• submissions that are acknowledged but not durably recorded;
• queue age or dispatch failures by eligible environment;
• pending workloads by scheduling reason;
• restart loops and repeated execution attempts;
• failed checkpoints or artifact uploads;
• control-plane database, queue, and storage saturation;
• telemetry gaps that make an important path unobservable; and
• recovery time or error-budget burn beyond the defined threshold.

Automation should act only on signals with clear semantics. Restarting an unhealthy process can help; repeatedly resubmitting a workload with an invalid image cannot. Preserve the original error and bound automated attempts.

Plan maintenance as a reliability event

Upgrades, certificate rotation, node replacement, storage migration, and policy changes exercise the same failure paths as incidents. Define compatibility windows and rollback conditions before starting.

Drain worker nodes without abandoning application state. Protect critical services with appropriate replica placement and disruption budgets, while remembering that a PodDisruptionBudget limits voluntary disruption rather than guaranteeing capacity or application correctness.

For platform upgrades, verify schema migrations, controller compatibility, queued-operation handling, and the ability to reconcile workloads created by the earlier version. Preserve a backup and a tested restore path that matches the upgrade procedure.

Apply the reliability model with Polyaxon

Polyaxon separates tracked operations and orchestration metadata from the Kubernetes workloads that execute them. In multi-environment commercial deployments, agents manage configured namespaces or clusters, while queues apply routing, priority, concurrency, resource, and cost controls. See agents and queues.

Use connections for artifact stores, registries, repositories, databases, and other dependencies. Scope credentials to the workload and make the recovery environment capable of resolving the same logical inputs and outputs. Persist required results through configured artifact storage rather than node-local paths.

Apply termination policies to bound retries, execution time, and cleanup. The application remains responsible for checkpoint semantics and safe external actions. Use run status, logs, metrics, and artifacts to connect infrastructure behavior with the operation outcome; use platform observability for the health of the Polyaxon deployment itself.

Keep product, cluster, and provider responsibilities explicit. Polyaxon can retain and route the operation; Kubernetes reconciles workload resources; storage and database systems protect their own data; the application determines whether a resumed computation is valid.

Practice recovery with representative work

Run controlled exercises against a non-production or safely bounded environment. Include an API replica loss, database failover or restore, unavailable artifact store, exhausted node pool, registry failure, node interruption during checkpointing, lost agent connectivity, and an ineligible fallback region.

For each exercise, capture detection time, user-visible effect, automation taken, manual decisions, recovered state, data loss if any, and time to an accepted result. Update the runbook and the dependency owner from what actually happened.

Reliable self-hosted ML infrastructure is not infrastructure that never fails. It is infrastructure that preserves accepted work, contains failures, explains what changed, and restores the complete user workflow within a tested objective.