Kubernetes container lifecycle hooks

Kubernetes container lifecycle hooks let a container run logic shortly after creation or during managed termination. They are useful for narrow, local coordination, but they are not a durable event system and cannot make every shutdown graceful.

Use hooks only after defining what the application itself must guarantee when processes start, receive a termination signal, or restart unexpectedly.

Know the two hooks

Kubernetes defines two container lifecycle hooks:

Hook	When it runs	Typical purpose
`PostStart`	Immediately after a container is created	Small local setup or notification tied to container creation
`PreStop`	Before managed container termination, within the Pod's grace period	Stop accepting work, drain, flush, or checkpoint before the termination signal

The container lifecycle hook documentation is careful about ordering and delivery. PostStart can run concurrently with the container's entrypoint, so there is no guarantee that it runs first. Hook delivery is intended to be at least once, which means handlers should tolerate a rare duplicate invocation. If the kubelet restarts at the wrong moment, a hook may also be missed.

That behavior is appropriate for lifecycle coordination, not for recording the only copy of an important business event.

Keep PostStart small

A PostStart handler blocks Kubernetes container management until it completes, but the main process may already be running. If the application must not serve before initialization finishes, use an init container, application startup logic, and a startup or readiness probe instead.

Good PostStart work is local, bounded, and repeatable. Avoid remote database migrations, irreversible API calls, or a long model download. If the hook fails, Kubernetes terminates the container, which can create a restart loop and repeat any partial side effects.

For ML services, model loading usually belongs in the application lifecycle with a startup probe that reports when the server is actually initialized.

Budget PreStop inside termination

PreStop runs after Pod termination begins and must finish before Kubernetes sends the container's termination signal. The termination grace-period clock is already running. If the hook plus application shutdown exceeds that budget, Kubernetes eventually forces termination.

This example gives the container up to 90 seconds for the hook and process shutdown together:

spec:
  terminationGracePeriodSeconds: 90
  containers:
    - name: model-server
      image: registry.example.com/model-server@sha256:REPLACE_WITH_DIGEST
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "/app/drain.sh"]

The command runs inside the container, and its resource use counts against that container. Keep the script in the image, make repeated execution safe, impose internal timeouts, and emit clear logs.

Coordinate readiness and shutdown

A common serving sequence is:

termination begins;
the application or hook marks the instance unavailable for new work;
readiness changes and traffic routing converges;
in-flight requests drain;
the main process handles its termination signal and exits.

A fixed sleep can create time for endpoint propagation, but it does not prove that traffic drained. Prefer application-aware shutdown that stops accepting work, tracks active requests, and exits when they complete or the internal deadline is reached.

Test the whole path with the actual ingress, Service, sidecars, and load balancer. The effective drain time depends on more than the container.

Do not rely on PreStop for every exit

PreStop is not invoked when a container has already terminated or completed. It also cannot run after abrupt node loss, process crash, kernel failure, or an immediate force deletion. A training job that protects its only checkpoint in a PreStop hook can still lose progress.

Durable ML workloads need application-level checkpoints written during normal execution. The handler can request a final checkpoint, but the application should make writes atomic, version them, and tolerate interruption. Store valuable outputs through configured artifact connections, not only on node-local storage.

Choose the correct mechanism

Use the narrowest primitive that matches the requirement:

Requirement	Better fit
Complete setup before app processes start	Init container
Delay traffic until initialization completes	Startup and readiness probes
Respond to `SIGTERM`	Application signal handler
Drain before managed termination	`PreStop` plus application shutdown
Persist job progress	Periodic application checkpointing
Record a durable lifecycle event	External idempotent event or state system
Clean up finished Jobs	Job/controller lifecycle and TTL policy

Observe and test hooks

Hook stdout is not exposed as ordinary container logs in the same way as the main process, so log meaningful state from the application and inspect Pod events when a hook fails. Track termination duration, forced kills, unfinished requests, and checkpoint outcomes.

Exercise deletion, rollout, eviction, process crash, and node-loss scenarios separately. Only managed termination paths invoke PreStop; your recovery design must also cover the paths that do not.

In Polyaxon, apply lifecycle and termination settings through versioned workload configuration or managed presets. Keep the operation's timeout and retry policy aligned with the container's behavior. Hooks should improve a well-defined lifecycle, not become hidden scripts that carry the workload's most important guarantees.