Keep long-running workloads authenticated through token rotation

A training job reads data successfully at startup, then receives authentication failures an hour later. The Pod is healthy and network access still works. Its client loaded a short-lived token once and kept using it after Kubernetes had rotated the file on disk.

Projected ServiceAccount tokens have a bounded lifetime and an intended audience. The kubelet renews the projected token, but the application must reload it. Rotation at the storage layer and refresh inside a client are separate steps. ServiceAccount token projection.

For a long-running notebook or inference service, correct refresh behavior is part of the workload's reliability contract. The broader Kubernetes Secrets guide covers identity and delivery choices; this article follows the token through a running process.

Define the trust relationship first

An external service does not accept a Kubernetes token merely because it is signed. The recipient must be configured to trust the issuer, validate the token and intended audience, and authorize its subject for a specific operation. A ServiceAccount identity is not permission to read every dataset.

For a custom service, Kubernetes documents TokenReview and issuer discovery as validation approaches. Offline validation and API-backed validation also have different implications for observing object deletion and revocation. Choose the method with the service owner rather than implementing a decoder and treating decoded claims as trusted. ServiceAccount authentication.

Keep two credentials distinct if a cloud or internal identity service exchanges the projected token for another credential. Refreshing the Kubernetes token file does not automatically refresh an already issued access token cached by the application's SDK.

Mount an audience-specific token

This example assumes an existing ml-team namespace and a dataset service configured for the example audience datasets.example.com. Replace that audience with the recipient's actual identifier. The ServiceAccount receives no Kubernetes RoleBinding here; creating it does not grant dataset access by itself.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: dataset-reader
  namespace: ml-team
automountServiceAccountToken: false
---
apiVersion: v1
kind: Pod
metadata:
  name: projected-token-demo
  namespace: ml-team
spec:
  serviceAccountName: dataset-reader
  automountServiceAccountToken: false
  restartPolicy: Never
  containers:
    - name: client
      image: python:3.12-slim
      command: ["python", "-c", "import time; time.sleep(7200)"]
      resources:
        requests: {cpu: "100m", memory: "64Mi"}
        limits: {cpu: "500m", memory: "128Mi"}
      volumeMounts:
        - name: dataset-identity
          mountPath: /var/run/dataset-identity
          readOnly: true
  volumes:
    - name: dataset-identity
      projected:
        sources:
          - serviceAccountToken:
              path: token
              audience: datasets.example.com
              expirationSeconds: 3600

Save this as projected-token-demo.yaml and apply it in an approved trial environment if you want to inspect the mount. The sleeping process does not call the dataset service or validate the trust relationship. The image tag is for the demonstration; pin a digest for maintained tooling.

Disabling automatic mounting does not prevent this explicitly requested projection. The token appears at /var/run/dataset-identity/token; the requested lifetime is subject to API-server configuration. Mount the directory directly, as shown: projected volumes mounted through subPath do not receive updates. Projected volume documentation.

Make the client reopen the file

For a custom HTTP client, this small helper illustrates the refresh boundary:

from pathlib import Path


def dataset_authorization_headers() -> dict[str, str]:
    token = Path('/var/run/dataset-identity/token').read_text().strip()
    if not token:
        raise RuntimeError('Dataset identity token is unavailable')
    return {'Authorization': f'Bearer {token}'}

Call the helper when constructing each request to the trusted dataset service over TLS, rather than once when initializing a long-lived session. Do not print the returned headers. The snippet deliberately does not implement a dataset client, retry loop, or token-exchange protocol.

For high request rates, prefer the supported authentication provider in your client library or a bounded refresh cache designed for the token's lifetime. The essential behavior is reopening the current projected path when refreshing. Holding an old file descriptor or copying the token into an environment variable can keep the process on old content.

Authentication failures also need bounded handling. Reloading once may recover a race with rotation; repeatedly retrying an audience mismatch or revoked permission will not.

Verify rotation without collecting secrets

Use an approved workload that runs across a renewal boundary and record the recipient's authentication outcomes. Keep tokens, Authorization headers, and token-bearing error dumps out of logs and run artifacts.

These are diagnostic possibilities, not a guarantee that every 401 response has one cause. The recipient's redacted audit evidence is more useful than exposing the bearer token in a support bundle.

For the learning fixture, remove the demo Pod when finished, then remove the newly created dataset-reader ServiceAccount only if nothing else uses it. In a real rollout, preserve the platform-owned ServiceAccount and its trust configuration.

Configure Polyaxon identity deliberately

Polyaxon lets a platform team select a ServiceAccount per workload and reuse it through a scheduling preset. A training job reading data, a notebook querying an external service, and an inference workload fetching model files can each use the identity appropriate to that task. This keeps the identity configuration with the workload definition instead of requiring each author to reconstruct it.

Polyaxon connections organize access to the external systems those workloads need. Pair the chosen ServiceAccount and supported volume configuration with the provider's workload-identity setup and a client that refreshes credentials. Inspect the rendered Pod for the intended audience, token mount, and consumer container, while retaining the permissions needed by Polyaxon's auxiliary containers.

Use a Polyaxon qualification run that lasts through credential rotation before adopting an image for long-running training or serving. Record request outcomes and refresh failures without logging tokens. Kubernetes and the provider issue credentials, the application client renews its access, and Polyaxon retains the workload configuration and qualification results. Keep that workload identity separate from Polyaxon API authentication.