Use kubectl exec safely for ML workload debugging

kubectl exec starts a process inside an existing container and connects its input and output to your terminal. It is useful for inspecting a running ML workload, but it is not SSH to a durable server and it should not become a deployment mechanism.

Use it after collecting external evidence, target the exact container explicitly, and leave the running filesystem unchanged. A manual repair disappears when Kubernetes replaces the Pod and makes the incident harder to reproduce.

Inspect from outside first

Before entering a container, capture the object state, events, and logs:

kubectl describe pod model-api-7d9b6f8d6d-k4m2p \
  --context acme-staging \
  --namespace ml-serving

kubectl logs model-api-7d9b6f8d6d-k4m2p \
  --context acme-staging \
  --namespace ml-serving \
  --container model-api \
  --timestamps

These commands often reveal image, scheduling, volume, probe, or application failures without interactive access. They also preserve evidence before an operator changes process or filesystem state.

Use Polyaxon run logs and metadata to confirm the operation, image, parameters, resources, and inputs represented by the Pod.

Understand the exec command

The official Kubernetes guide to getting a shell in a running container explains the command boundary:

kubectl exec model-api-7d9b6f8d6d-k4m2p \
  --context acme-staging \
  --namespace ml-serving \
  --container model-api \
  --stdin \
  --tty \
  -- /bin/sh

The double dash separates kubectl arguments from the command executed in the container. --stdin forwards input and --tty allocates an interactive terminal.

Specify --container even when the Pod currently has only one application container. That makes the target clear and prevents a later sidecar from changing which container receives the command.

Prefer one diagnostic command

An interactive shell is not always necessary. Run the smallest read-only command that answers the question:

kubectl exec model-api-7d9b6f8d6d-k4m2p \
  --context acme-staging \
  --namespace ml-serving \
  --container model-api \
  -- cat /proc/1/status

Useful checks include process state, mounted filesystems, DNS configuration, listening sockets, environment variable names, and access to a required local path.

Avoid printing secret values. Terminal output can be recorded in shell history, support transcripts, CI logs, or screen recordings.

Know what exec cannot diagnose

kubectl exec requires a running container and an executable inside it. It cannot enter a container that never started because of an image pull, mount, admission, or scheduling failure.

Minimal and distroless images may not include a shell, package manager, curl, ps, or network tools. That is a security and supply-chain benefit, not an image defect.

Exec also shows one container's view. It does not explain the complete Service path, node routing, cloud load balancer, storage backend, scheduler decision, or another distributed-training worker.

Use ephemeral containers for minimal images

Kubernetes supports ephemeral debugging containers when the application image lacks tools or ordinary exec is insufficient.

An approved example is:

kubectl debug pod/model-api-7d9b6f8d6d-k4m2p \
  --context acme-staging \
  --namespace ml-serving \
  --image=busybox:1.36 \
  --target=model-api \
  --stdin \
  --tty

Use an internally approved, digest-pinned debug image in production. Ephemeral containers expand the software and access present in the Pod and may be retained in its specification, so treat their creation as an audited privileged action.

For a crashing application or an unsafe-to-touch production Pod, create a debug copy according to the cluster runbook instead of changing the original workload.

Preserve immutability

Do not use exec to:

• install packages into the application container;
• patch source or model files;
• change production configuration;
• copy credentials into the filesystem;
• start an unmanaged replacement process;
• turn a one-off fix into hidden persistent state.

If a file, package, command, or configuration must change, rebuild the image or update the declared operation and roll out a new version.

A temporary diagnostic action should be read-only wherever possible. If an emergency mutation is unavoidable, record the exact command, affected run, owner, reason, and follow-up change in the source-controlled layer.

Limit and audit access

Kubernetes RBAC treats Pod exec as access to the pods/exec subresource. Grant it only to roles that need interactive workload access and separate it from ordinary log viewing.

The effective power depends on the container. Shell access to a privileged Pod, mounted service-account token, host path, dataset credential, or production model store can become broader infrastructure access.

Use short-lived human identity, require an incident or support reason, retain API audit logs, and review access to sensitive namespaces. Network policy does not make an authorized shell harmless.

Debug distributed ML carefully

For a distributed job, choose the replica and container from evidence. Compare environment, mounts, network reachability, and process state with a healthy peer without exposing secrets.

Do not restart one worker casually. Collective training can leave the remaining workers blocked while they continue consuming accelerators, and a framework may require the entire group to restart from a checkpoint.

For model serving, avoid running heavy diagnostics in the request path. A CPU, memory, disk, or network probe can create the latency incident it is meant to investigate.

Prefer Polyaxon-native access where appropriate

Polyaxon provides run-scoped logs, metadata, artifacts, and operation shell commands that keep the investigation connected to the operation identity.

Use direct kubectl exec when Kubernetes-level access is genuinely required. Use the narrowest identity and preserve the link back to the Polyaxon run, project, cluster, namespace, image, and incident.

Interactive access is successful when it produces evidence and a reproducible fix—not when the container appears healthy only until its next restart.