Kubernetes probes for ML services

A model server may need minutes to download weights, allocate accelerator memory, compile kernels, and warm a cache. Once it is running, a temporary dependency outage may make it unable to serve traffic without making the process unrecoverable. One generic /health endpoint cannot represent all of those states safely.

Kubernetes has three probe types because it needs three different answers: has the application finished starting, should it receive traffic, and is restarting the container the right recovery action?

Give each probe one decision

The probe names describe the action Kubernetes takes, not merely a health score:

The Kubernetes probe documentation emphasizes that a poorly designed liveness probe can create cascading failures. A dependency becoming slow is often a readiness problem, not proof that the local process must be killed.

Model the startup path explicitly

ML services often have a long but legitimate initialization sequence:

1. Pull the container image.
2. Fetch model and tokenizer artifacts.
3. Verify checksums and configuration.
4. Allocate CPU, memory, or accelerator resources.
5. Load weights and compile optimized execution paths.
6. Run a local smoke inference.
7. Mark the process ready for traffic.

Use a startup probe when this sequence can exceed the safe liveness window. The maximum allowance is approximately:

failureThreshold × periodSeconds

For example, 60 × 5 seconds gives the process up to five minutes to start. Choose the budget from observed cold starts at realistic model sizes and storage conditions, not from a fast developer laptop.

Configure distinct endpoints

The following Deployment uses a generous startup window, a responsive readiness check, and a conservative liveness check:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-server
  namespace: inference
spec:
  replicas: 3
  selector:
    matchLabels:
      app: model-server
  template:
    metadata:
      labels:
        app: model-server
    spec:
      containers:
        - name: server
          image: ghcr.io/example/model-server:1.4.0
          ports:
            - name: http
              containerPort: 8080
          startupProbe:
            httpGet:
              path: /startup
              port: http
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 60
          readinessProbe:
            httpGet:
              path: /ready
              port: http
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 3
            successThreshold: 1
          livenessProbe:
            httpGet:
              path: /live
              port: http
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 6

The endpoints should have deliberately different semantics:

• /startup succeeds after the model is loaded and the server can complete a local verification.
• /ready succeeds while the instance can accept new requests within its operating limits.
• /live succeeds while the process event loop and critical internal workers are making progress.

Keep probe handlers cheap, deterministic, and local. A health request should not perform inference, scan a large model, or wait on a chain of remote services.

Do not turn dependency failures into restart storms

Suppose the model server cannot reach a feature store. Marking the Pod unready can prevent new traffic until the dependency recovers. Failing liveness at the same time restarts the server, forcing it to reload the model and increasing pressure on the registry, object store, nodes, and remaining replicas.

Use readiness for temporary conditions such as:

• a required downstream service is unavailable;
• the request queue is beyond a safe threshold;
• the instance is draining;
• the model or adapter needed for traffic is not loaded;
• accelerator memory pressure makes new work unsafe.

Use liveness for local failures where restarting is likely to help, such as a deadlocked worker, a permanently wedged event loop, or a failed internal supervisor that cannot recreate required processes.

Choose the right probe mechanism

Kubernetes supports HTTP, TCP, command, and gRPC probes. Match the mechanism to what you need to prove:

• HTTP is usually the clearest option for web APIs and model servers.
• gRPC uses the standard gRPC health protocol and avoids adding an HTTP endpoint to a gRPC-only service.
• TCP proves that a port accepts a connection, but not that requests can be processed correctly.
• Exec can inspect local process state or files, but it creates a process inside the container on every check and is easy to make unnecessarily expensive.

A successful TCP handshake is not proof that a model is loaded. A successful HTTP response from a dependency gateway is not proof that the local server can make progress. Select the narrowest check that supports the decision.

Tune from failure budgets

Four values determine how quickly a probe reacts:

• initialDelaySeconds delays the first check.
• periodSeconds controls how often checks run.
• timeoutSeconds limits each attempt.
• failureThreshold controls consecutive failures before action.

Treat the combination as a recovery budget. A liveness probe every ten seconds with a threshold of six tolerates roughly one minute of consecutive failures before restart, excluding request timing. That may be appropriate for an expensive model load and too slow for a lightweight API.

Tune readiness more responsively than liveness so traffic stops before Kubernetes restarts the container. Add enough replicas, disruption controls, and rollout capacity that one unready Pod does not overload the rest.

Use different health semantics for batch workloads

Training, tuning, and batch evaluation Jobs are not traffic-serving replicas. A liveness restart can erase expensive in-memory progress or repeat non-idempotent work. For these workloads, process exit codes, checkpoints, retry policy, termination behavior, and application-level progress signals are usually more useful than an HTTP health endpoint.

Define termination and retry behavior around the workload's recovery model. If a training process can recover from a checkpoint, verify that checkpoints are recent and durable before making retries more aggressive.

Observe what probes cause

Probe configuration is incomplete without feedback. Track:

• container restart count and reason;
• readiness transitions;
• probe latency and timeout rate;
• cold-start duration by model and node type;
• time to first successful request;
• traffic and queue pressure on remaining replicas;
• artifact download and model-load duration.

Correlate Kubernetes events with application logs and infrastructure metrics. Polyaxon's platform observability guidance can sit alongside cluster telemetry so teams can distinguish an application failure from slow storage, node pressure, scheduling delay, or a dependency outage.

Roll out probes safely

1. Instrument startup, readiness, and progress before enabling restarts.
2. Measure healthy and degraded behavior under realistic load.
3. Add a startup probe for the complete cold-start path.
4. Add readiness and verify traffic is removed and restored correctly.
5. Add liveness only for known unrecoverable states.
6. Test dependency loss, overload, slow storage, and node pressure.
7. Alert on probe-driven restarts and sustained unready capacity.

The goal is not to make every endpoint green. It is to give Kubernetes enough accurate information to take the least disruptive recovery action.