Kubernetes RBAC for ML workloads

Most ML containers do not need to call the Kubernetes API. They need to read input data, use compute, write artifacts, and exit. Giving every workload a broad service account turns a compromised notebook, dependency, or training image into a cluster-administration problem.

Kubernetes role-based access control (RBAC) lets platform teams authorize specific actions for specific identities at a defined scope. The useful question is not whether RBAC is enabled. It is whether each human, controller, and workload can do exactly what it needs—and no more.

Separate platform permissions from cluster permissions

An ML platform usually has at least two authorization layers:

• Platform authorization controls who may view a project, start a run, use a connection, approve a model, or administer an organization.
• Kubernetes authorization controls which API resources an authenticated identity may read or change inside the cluster.

Passing one layer does not imply access in the other. A user who may start an experiment in Polyaxon should not automatically receive direct Kubernetes credentials. Similarly, the Kubernetes ServiceAccount used by a run should not inherit the submitting user's platform permissions.

Polyaxon provides organization, team, and project RBAC for the application layer. Kubernetes RBAC remains the enforcement layer for Kubernetes API access inside each compute environment. Treat both as explicit parts of the authorization path.

Build policy from four questions

Every Kubernetes RBAC decision can be reduced to four questions:

1. Who is acting? A user, group, or ServiceAccount.
2. What action is required? A verb such as get, list, watch, create, update, patch, or delete.
3. Which resource is affected? Pods, Jobs, Secrets, ConfigMaps, custom resources, or a subresource such as pods/log.
4. Where is the permission valid? One namespace or the whole cluster.

A Role defines permissions within a namespace. A ClusterRole defines a reusable set of permissions that may include cluster-scoped resources. A RoleBinding grants a Role or ClusterRole inside one namespace, while a ClusterRoleBinding grants a ClusterRole across the cluster. The Kubernetes RBAC reference documents these scope rules in detail.

Start namespaced. Cluster-wide access should be an exception justified by a real cluster-scoped responsibility.

Give each workload class its own identity

Do not bind workload permissions to the namespace's default ServiceAccount. Any Pod that omits serviceAccountName can use that identity, which silently expands the blast radius.

Instead, create a dedicated ServiceAccount for each workload class that needs Kubernetes API access. The following example lets an inventory task read Pod metadata and logs in ml-team-a without changing workloads or reading Secrets:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: experiment-reader
  namespace: ml-team-a
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: experiment-reader
  namespace: ml-team-a
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: experiment-reader
  namespace: ml-team-a
subjects:
  - kind: ServiceAccount
    name: experiment-reader
    namespace: ml-team-a
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: experiment-reader

If a container does not call the Kubernetes API, avoid mounting an API token where your workload and platform configuration permit it. Fewer credentials are safer than broader credentials with good intentions.

Attach the identity at the operation boundary

Polyaxon operations use the platform's configured ServiceAccount by default. When a particular component needs a different identity, declare it in the operation environment instead of modifying a shared default:

version: 1.1
kind: component
name: inspect-running-pods
run:
  kind: job
  environment:
    serviceAccountName: experiment-reader
  container:
    image: bitnami/kubectl:latest
    command: ["kubectl"]
    args: ["get", "pods", "--namespace", "ml-team-a"]

The Polyaxon service-account guide explains per-operation and preset-based configuration. When Polyaxon auxiliary containers are enabled, ensure the custom ServiceAccount also retains the minimum permissions those containers require.

Use a preset when the same identity applies to a governed workload class. This makes the policy reusable without allowing each component author to choose an arbitrary privileged identity.

Test effective permissions, not YAML intent

Policy review should include the permission Kubernetes will actually evaluate after all bindings are combined. Use kubectl auth can-i with an explicit namespace and identity:

kubectl auth can-i list pods \
  --namespace ml-team-a \
  --as system:serviceaccount:ml-team-a:experiment-reader

kubectl auth can-i get pods/log \
  --namespace ml-team-a \
  --as system:serviceaccount:ml-team-a:experiment-reader

kubectl auth can-i delete jobs.batch \
  --namespace ml-team-a \
  --as system:serviceaccount:ml-team-a:experiment-reader

The first two checks should return yes; the last should return no. Add positive and negative checks to policy review so a role is tested for both required access and prohibited access.

For a broader inspection, list the effective permissions:

kubectl auth can-i --list \
  --namespace ml-team-a \
  --as system:serviceaccount:ml-team-a:experiment-reader

Treat some permissions as privilege boundaries

Some permissions are more powerful than their resource names suggest:

• Reading Secrets may expose database, registry, storage, or cloud credentials.
• Creating or patching Pods can allow an identity to run code under another ServiceAccount or mount sensitive volumes.
• Using pods/exec allows commands inside running containers.
• Creating RoleBindings or ClusterRoleBindings may enable privilege escalation.
• The bind, escalate, and impersonate verbs deliberately cross authorization boundaries.
• Changing admission webhooks, custom resource definitions, nodes, or namespace policy has cluster-wide consequences.

Avoid wildcard resources and verbs. New API resources and subresources can make wildcard grants broader over time without a visible policy change. The Kubernetes RBAC good-practices guide also recommends least privilege, namespace-level grants where possible, and avoiding unnecessary token access.

Keep authorization reviewable

Store Roles and bindings as versioned manifests. Give each binding an owner, purpose, and review date. Separate human access from controller and workload access, and prefer short-lived authentication for people and automation.

Audit changes to these objects:

• roles and clusterroles
• rolebindings and clusterrolebindings
• serviceaccounts
• token and credential issuance
• impersonation requests

Review unused bindings and stale identities regularly. Removing access when a project, integration, or workload class is retired is part of the lifecycle—not an optional cleanup task.

A practical rollout sequence

1. Inventory which workloads actually call the Kubernetes API.
2. Group them by required resource, verb, namespace, and operational owner.
3. Create a dedicated ServiceAccount and namespaced Role for each group.
4. Bind only the required permissions.
5. Verify expected yes and no results with kubectl auth can-i.
6. Attach the identity through a reviewed Polyaxon component or preset.
7. Observe authorization failures and audit events during rollout.
8. Remove broad legacy bindings after the narrow policy is proven.

Good Kubernetes RBAC should be uneventful. Workloads complete, controllers reconcile, and operators investigate incidents without anyone carrying more authority than the task requires.