Rotate workload TLS with Kubernetes Pod certificates

A notebook may stay open for days. A model server can outlive several credential lifetimes. If either loads a TLS certificate only at startup, renewing a file on disk is not enough to keep new connections working.

Kubernetes Pod certificates provide a native path for requesting a workload's private key and certificate chain, then refreshing them through a projected volume. Pod certificates and ClusterTrustBundles became stable in Kubernetes 1.37. The application still owns loading those credentials and applying them to its TLS connections. Kubernetes 1.37 announcement.

For ML workloads, this can support training clients that authenticate to a dataset service, notebooks calling internal APIs, and inference services using mTLS. It requires a compatible certificate issuer and a receiving service that understands the issued identity.

Establish the issuer and the recipient

The certificate projection names a signer. That signer determines whether a request is allowed and what identity the certificate represents. Kubernetes does not turn an arbitrary signerName into a functioning certificate authority. The 1.37 announcement explicitly notes that a third-party signer is needed; the upstream Tinycert example is a learning implementation, not a production recommendation.

Before configuring a workload, agree on three things with the service owner:

A successful certificate handshake does not by itself authorize every dataset operation. Likewise, the client must still validate the server's identity, including its expected DNS name when using ordinary HTTPS hostname verification. Authentication and application authorization need to agree.

This mechanism complements the existing TLS and mTLS guide. It supplies workload credentials; it does not automatically install a service mesh, configure a model server's TLS listener, or replace the site's ingress certificate.

Project credentials and trust into a Pod

The example below assumes a Kubernetes 1.37 cluster and three resources prepared by the platform team:

• An existing ml-team namespace.
• An installed signer named ml.example.com/workloads that permits the dataset-reader ServiceAccount and issues suitable client credentials.
• A ClusterTrustBundle named dataset-server-roots containing the CA certificates needed to verify the dataset server.

The names are placeholders for your installation. Merely applying this manifest does not install the signer or create the trust bundle. The native Pod is a small projection example that sleeps so its mounted files can be inspected; it is not a dataset client or a deployed inference endpoint.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: dataset-reader
  namespace: ml-team
automountServiceAccountToken: false
---
apiVersion: v1
kind: Pod
metadata:
  name: workload-certificate-demo
  namespace: ml-team
spec:
  serviceAccountName: dataset-reader
  automountServiceAccountToken: false
  restartPolicy: Never
  containers:
    - name: inspect
      image: debian:bookworm-slim
      command: ["sleep", "infinity"]
      resources:
        requests:
          cpu: 10m
          memory: 32Mi
        limits:
          cpu: 100m
          memory: 64Mi
      volumeMounts:
        - name: workload-tls
          mountPath: /var/run/workload-tls
          readOnly: true
  volumes:
    - name: workload-tls
      projected:
        defaultMode: 0400
        sources:
          - podCertificate:
              signerName: ml.example.com/workloads
              keyType: ECDSAP256
              maxExpirationSeconds: 86400
              credentialBundlePath: client.pem
          - clusterTrustBundle:
              name: dataset-server-roots
              path: server-roots.pem

The kubelet generates the private key; the signer issues the certificate. The Pod requests a certificate lifetime of at most one day, and the signer can issue a shorter one. client.pem contains the private key and certificate chain together. server-roots.pem contains trust anchors for verifying the remote server. The server needs its own configuration to trust the client's issuing CA. Projected volume documentation.

The restrictive file mode here suits the root process in this inspection image. For a non-root application, arrange ownership and permissions for its actual UID and GID through your supported Pod security configuration. Mount the directory only into containers that need the credentials, and keep it out of workspace exports and artifact collection.

Treat rotation as an application event

Kubelet updates the projected files as credentials and trust bundles change. The application must pick up those updates. Prefer the combined credential bundle: reading separate key and certificate paths during a rotation can mix two generations. Mount the directory rather than an individual file through subPath, which does not receive projected updates. Projection and rotation behavior.

For your TLS library or proxy, establish the following behavior:

1. Detect a directory update or periodically reopen the bundle path.
2. Read the credential bundle as one snapshot and parse the key and chain from that snapshot.
3. Validate the new credentials, then replace the TLS configuration used for new connections.
4. Refresh server trust anchors when the trust file changes as well.
5. Report reload failures and certificate expiry without logging private key material.

Passing the same filename to two separate library reads is not necessarily a single snapshot. Check how the library actually loads certificates. If the application only supports startup-time loading, use a controlled restart strategy with sufficient overlap before expiry, and account for notebook state or inference capacity when doing so.

Existing connections and connection pools also need an explicit policy. Replacing a TLS configuration does not reauthenticate every connection already open. When verifying rotation, force a new connection after the change and establish that it uses the refreshed credential.

Apply the pattern to Polyaxon workloads

Polyaxon provides useful workload boundaries for this setup: select the agent and queue, choose the Kubernetes ServiceAccount, mount the required volumes, and retain the execution record. Certificate issuance remains with Kubernetes and the installed signer; TLS behavior remains with the application.

The relevant documented interfaces are custom ServiceAccounts and runtime volumes. They apply to jobs, services, and distributed workloads. For example, the workload identity is selected with this Polyaxonfile fragment:

run:
  kind: job
  environment:
    serviceAccountName: dataset-reader

This fragment selects an identity only; the credential volume and application configuration still need to be provided. The new podCertificate and clusterTrustBundle fields must be preserved by the Polyaxon and Kubernetes client versions used in your installation. Confirm that the resolved Pod contains them before depending on this native projection path. The general volume interface alone does not establish compatibility with every newly added Kubernetes field.

Where a platform already uses a reviewed admission integration to inject workload certificates, its owner can apply the projection to the selected workloads there. That integration and its signer policy are separate platform components, not functionality installed by the fragment above.

Three useful applications follow:

• Training: a GPU job authenticates to an internal data service throughout a long run. The data client reloads credentials as it reconnects; scheduling a GPU does not change the identity mechanism.
• Notebooks and sandboxes: an interactive environment keeps calling internal services after the initial certificate expires. Give the workload only the permissions intended for code running in that environment.
• Inference: a service authenticates to an upstream API, or presents server credentials when its TLS stack and signer support that use. Incoming server TLS needs the correct names, usages, listener configuration, and client trust in addition to the volume.

Capture certificate expiry, reload success, and connection errors as operational metrics. Link that evidence to the Polyaxon run or service, without collecting the credential files. This extends the same lifecycle reasoning used in the ServiceAccount token rotation article to X.509 credentials.

Follow failures through the issuance path

For the native example, these read-only commands show the Pod and certificate requests:

kubectl describe pod workload-certificate-demo -n ml-team

kubectl get podcertificaterequests -n ml-team

kubectl get clustertrustbundle dataset-server-roots

Have an authorized operator correlate the request with the Pod identity and inspect its status. A pending or denied request leads to the signer and its policy. A missing trust bundle leads to trust distribution. A mounted, valid certificate followed by a rejected request leads to the TLS peer's trust, identity checks, or authorization rules.

Before adopting the pattern for a long-lived service, exercise a complete refresh interval and a new connection using the replacement credential. Include a reload failure and an unavailable signer in the operational review. Successful initial issuance is the beginning of the lifecycle, not evidence that the running application handles rotation.

When you finish inspecting the example, remove its sleeping Pod:

kubectl delete pod workload-certificate-demo -n ml-team

Remove the example ServiceAccount only if you created it for this exercise and no other workload uses it. Keep shared signers and trust bundles under the platform owner's lifecycle policy.