Kubernetes startup probes for model servers

A model server can be healthy while taking several minutes to become usable. It may download weights, validate artifacts, allocate accelerator memory, compile kernels, and run a warmup inference before it can answer traffic.

If liveness checking starts too early, Kubernetes can repeatedly restart a process that only needs more initialization time. A startup probe creates a separate budget for that phase without making liveness permanently lenient.

Understand the startup gate

When a startup probe is configured, Kubernetes holds liveness and readiness checks until startup succeeds. If startup failures reach the configured threshold, the kubelet restarts the container according to its restart policy. The Kubernetes probe documentation defines the behavior and supported mechanisms.

This produces three distinct decisions:

A startup probe does not replace readiness. The first successful startup check should mean initialization is complete; readiness can still turn on and off as the serving state changes.

Measure cold starts first

Do not choose thresholds from intuition. Measure cold starts under representative conditions:

• uncached image pull;
• uncached model and tokenizer download;
• the largest supported model variant;
• expected object-store latency;
• accelerator allocation and runtime initialization;
• kernel compilation or optimization;
• local smoke inference;
• credential or dependency retries.

Separate image-pull and scheduling delays from time inside the running container. Kubernetes probes begin after the container starts, so a five-minute image pull does not consume a startup probe's five-minute budget.

Use a high percentile plus deliberate headroom. A narrow threshold that works only with a warm node cache will create restart loops during rollout or recovery.

Calculate the failure window

The approximate startup allowance is:

failureThreshold × periodSeconds

This example allows up to ten minutes after startup checks begin:

startupProbe:
  httpGet:
    path: /startup
    port: http
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 120

initialDelaySeconds can postpone the first check, but a startup probe that begins early and tolerates expected failures usually gives better evidence about progress. Keep timeoutSeconds long enough for the local check but short enough to detect a stuck handler.

Make the endpoint local and specific

The startup endpoint should answer whether this process completed its required initialization. It might verify that:

• model artifacts passed integrity checks;
• the runtime loaded the requested model;
• required accelerator memory was allocated;
• a local warmup execution succeeded;
• the serving loop is able to accept work.

Avoid making startup depend on every optional remote system. A telemetry backend outage should not necessarily prevent a model from starting. Conversely, if the model cannot operate without a required artifact or configuration, reporting startup success too early only moves the failure into live traffic.

Keep the endpoint cheap. A full inference on every five-second probe can compete with initialization or serving work. Perform the expensive verification once, retain the result in process state, and let the probe read that state.

Choose the right probe mechanism

Kubernetes supports HTTP, TCP, gRPC, and exec probes. Prefer a protocol-level health endpoint when the serving process already exposes one. TCP only proves that a connection can be opened; it does not prove the intended model loaded. Exec probes can inspect local state but add process overhead and depend on utilities inside the image.

Whichever mechanism you choose, make its contract testable outside Kubernetes. Operators should know what failure means and which initialization stage is blocking.

Preserve useful failure evidence

When the startup threshold is exceeded, capture enough evidence to distinguish a slow start from a permanent error:

• stage name and elapsed time;
• artifact identity and size, without credentials;
• retry count and last dependency error;
• accelerator/runtime initialization errors;
• container restart count and previous logs;
• node, image, model version, and configuration.

Polyaxon run metadata connects those details to the operation that launched the service. Export durable logs and status before repeated restarts rotate away the useful context.

Test rollout and recovery paths

Exercise the slow paths intentionally: empty caches, delayed object storage, invalid artifacts, insufficient accelerator memory, and a warmup that never completes. Confirm that permanent failures eventually stop restarting according to the wider deployment strategy and surface a clear reason.

Also test termination. Model servers may need time to stop accepting traffic and finish in-flight work; startup configuration does not cover graceful shutdown.

For the relationship among all three probe types, see Kubernetes probes for ML services. The narrow principle for startup probes is simple: give legitimate initialization enough measured time, but keep a finite boundary for initialization that will never succeed.