Manage Kubernetes Secrets for ML workloads

ML workloads often need object-storage credentials, registry authentication, source-control tokens, database passwords, signing keys, and API credentials. Kubernetes Secrets keep those values out of container images and ordinary Pod fields, but a Secret object is not automatically safe.

Base64 is an encoding, not encryption. Security depends on storage encryption, access control, workload identity, delivery, rotation, and the behavior of every process that receives the value.

Start with a threat model

Identify who must use each credential, which system issues it, how long it remains valid, and what an attacker could do with it. Separate credentials by environment, team, workload purpose, and privilege.

A training job that reads one dataset should not inherit a platform administrator token. A notebook should not receive every production connection available in its namespace. Prefer short-lived, scoped credentials tied to workload identity when the external service supports them.

Encrypt Secret data at rest

Kubernetes stores API data in etcd. Secret values are not encrypted at rest by default unless the cluster is configured to do so. Enable an API-server encryption provider, protect encryption keys separately, and define a rotation procedure.

Encryption at rest protects stored data; it does not prevent an authorized API client, admitted Pod, or compromised node from reading a Secret in use. The Kubernetes Secrets documentation recommends encryption, least-privilege RBAC, restricted container access, and external secret stores where appropriate.

Backups of etcd and control-plane data require the same protection and recovery controls as the live store.

Treat workload creation as secret access

RBAC permission to get a Secret is obviously sensitive. Permission to list or watch Secrets can expose their contents as well. Less obviously, permission to create Pods in a namespace can allow a user to mount and read Secrets available there.

Design namespaces, ServiceAccounts, admission policy, and RBAC together. Check effective authorization without retrieving any Secret value:

kubectl auth can-i get secret/object-store-reader \
  --context acme-production \
  --namespace ml-team \
  --as system:serviceaccount:ml-team:trainer

kubectl auth can-i create pods \
  --context acme-production \
  --namespace ml-team \
  --as system:serviceaccount:ml-team:trainer

The RBAC good-practices guide documents these escalation paths.

Prefer workload identity over static keys

Cloud and internal services can often exchange a Kubernetes ServiceAccount identity for a short-lived credential. This removes a long-lived key from the namespace and gives the provider a clearer audit trail.

Bind identity to a dedicated ServiceAccount, restrict which Pods can use it, constrain the external role, and validate issuer, audience, and subject. A default ServiceAccount shared by unrelated workloads is a weak identity boundary.

When static material is unavoidable, use an approved external secret manager or synchronization mechanism. Document whether the Kubernetes Secret is the source of truth, a cache, or a delivery object.

Choose file or environment delivery

Secrets can be exposed as mounted files or environment variables. Environment variables are resolved when a container starts and are easy for application code to consume, but they can appear in process diagnostics, crash reports, or accidental environment dumps.

Mounted Secret volumes can update eventually when the Secret changes. The application must reread the file; an open file descriptor or startup-only configuration will not rotate automatically. subPath mounts do not receive automatic Secret volume updates.

Deliver a Secret only to the container that needs it. A sidecar in the same Pod does not require every credential mounted by the main container.

Avoid accidental disclosure

Do not place Secret values in:

• image layers or Docker build arguments;
• Git history or generated manifests;
• command-line arguments visible in process listings;
• Pod labels or annotations;
• experiment parameters, logs, metrics, or exception messages;
• copied debugging output and support tickets.

Keep kubectl get secret -o yaml, decoding commands, and shell tracing out of routine runbooks. Inspect names, types, references, permissions, and events before requesting access to the value itself.

Design rotation and revocation

Rotation is a workflow, not only an update. Define how a new credential is issued, delivered, observed by the application, validated, and followed by revocation of the previous one.

For workloads that read credentials only at startup, coordinate a controller rollout or a new operation after the Secret changes. For long training runs, prefer renewable credentials or an identity-based client that can refresh without discarding progress.

Monitor authentication failures and credential age without exporting values. Test emergency revocation and recovery.

Use Polyaxon connections

Polyaxon connections let platform teams define approved access to artifacts, data, registries, and external services, then expose those connections only to eligible operations. Workload authors reference a connection rather than copying credentials into a component.

Keep connection ownership, target environment, allowed projects, underlying identity, rotation path, and audit expectations explicit. Polyaxon reduces repeated secret handling; Kubernetes and the external provider still enforce the security boundaries.