Kubernetes sidecars for ML workloads

A sidecar is a helper container that shares a Pod's lifecycle, network namespace, and optionally its volumes. ML workloads may use sidecars for a local proxy, credential refresh, artifact synchronization, or another capability that must remain colocated with the main process.

Colocation is a strong coupling decision. Containers in one Pod schedule, scale, restart, and consume resources as a unit, so use a sidecar only when that coupling is intentional.

Distinguish native and regular sidecars

Historically, teams placed two regular containers in spec.containers and treated one as a sidecar by convention. Kubernetes now supports native sidecar containers as restartable init containers with restartPolicy: Always.

Native sidecars provide useful lifecycle semantics:

• they start in init-container order before application containers;
• a startup probe can gate later initialization;
• they keep running for the Pod's lifetime;
• they do not prevent a Job from completing after its main container exits;
• they terminate after the main application containers, in reverse declaration order.

Confirm the destination cluster supports the sidecar container API before using it. A manifest cannot assume the feature exists in every older Kubernetes environment.

Use sidecars for tight coupling

A sidecar can be appropriate when the helper must:

• share a Unix socket or localhost-only endpoint;
• read or write the same bounded volume;
• start before the application and remain available throughout execution;
• be deployed and versioned with one workload;
• scale exactly with each Pod replica.

If many workloads call the same helper and independent scaling is useful, make it a Service instead. If the work finishes before the application starts, use an ordinary init container. If it should run once per node, consider a DaemonSet.

Configure startup explicitly

This example starts an artifact helper before a trainer and shares a workspace:

apiVersion: v1
kind: Pod
metadata:
  name: training-with-helper
  namespace: ml-team
spec:
  initContainers:
    - name: artifact-helper
      image: registry.example.com/artifact-helper@sha256:REPLACE_WITH_DIGEST
      restartPolicy: Always
      startupProbe:
        exec:
          command: ["/app/healthcheck"]
        periodSeconds: 2
        failureThreshold: 30
      volumeMounts:
        - name: workspace
          mountPath: /workspace
  containers:
    - name: trainer
      image: registry.example.com/trainer@sha256:REPLACE_WITH_DIGEST
      volumeMounts:
        - name: workspace
          mountPath: /workspace
  volumes:
    - name: workspace
      emptyDir:
        sizeLimit: 20Gi

Replace placeholders and choose persistent or remote storage if data must survive Pod loss. A shared emptyDir is still node-local and ephemeral.

Budget resources for the whole Pod

Sidecars consume CPU, memory, storage, and sometimes network throughout the workload. Declare requests and limits based on measured behavior. A log or artifact helper with an unbounded buffer can cause the trainer to be throttled or the Pod to be evicted.

Scheduling accounts for init and sidecar resources according to Kubernetes' Pod resource rules. Inspect the effective Pod request in the cluster version you operate, especially when a sidecar has a large startup peak.

Do not add a sidecar to every workload to avoid operating a shared collector. Per-Pod helpers multiply resource and rollout cost with workload scale.

Define data ownership

Two containers writing one volume need a protocol. Specify file naming, atomic completion, permissions, ownership, retry behavior, and cleanup. A helper should not upload a checkpoint while the trainer is still writing it.

Use temporary names and atomic rename where the filesystem supports it, or publish a manifest only after all files are complete. Make uploads idempotent and include the run and artifact version in the destination identity.

Polyaxon artifact connections provide a durable destination, while mounted volumes provide the local exchange path.

Limit the security boundary

Containers in a Pod share a network namespace and may share volumes. A proxy or collector can observe sensitive application traffic or data if given broad access. Assign the Pod a scoped service account, mount credentials only where needed, use non-root security contexts, and avoid a privileged sidecar as a generic integration solution.

If the helper needs materially different trust, ownership, or network policy, it may not belong in the same Pod.

Test lifecycle failures

Exercise sidecar startup failure, restart, slow shutdown, full buffers, unavailable destinations, and main-container completion. Verify that Jobs finish, serving Pods drain, important data persists, and retries do not duplicate outputs.

Observe each container separately while keeping the Pod-level outcome visible. A healthy trainer with a failed artifact helper may still produce an unusable run; a healthy helper cannot rescue a trainer that never completed.

Use sidecars to express genuine colocation, not as a default architecture pattern. The best sidecar has a narrow contract, bounded resources, compatible lifecycle, and a failure mode that remains understandable from the Polyaxon run and Kubernetes Pod evidence.