Sandbox AI agents on Kubernetes
Design isolated Kubernetes execution for AI agents with a threat model, stronger runtimes, scoped identity, controlled egress, bounded storage, and auditable cleanup.
An agent that executes generated code is not only running a containerized application. It is interpreting untrusted instructions while holding whatever authority the process can reach: files, credentials, network destinations, Kubernetes APIs, tool endpoints, and retained state.
Kubernetes provides controls for reducing that authority, but no single field turns a Pod into a safe sandbox. Start with a threat model, select an isolation level, and verify the complete execution boundary under the conditions the agent will encounter.
Define what the sandbox must contain
List the code and inputs that may be hostile or simply wrong. Include user prompts, retrieved documents, repository content, generated shell commands, downloaded packages, model-produced URLs, tool responses, and files left by earlier sessions.
Then list the effects that must be prevented or constrained:
- reading another tenant's data or credentials;
- modifying the host, runtime, or cluster control plane;
- reaching unauthorized internal or external services;
- persisting malware or poisoned state for a later task;
- exhausting CPU, memory, disk, processes, network, or accelerators;
- impersonating a user beyond the approved task; and
- hiding actions from the audit and evidence trail.
Assign a trust level to each workload class. A trusted engineer's notebook, a first-party evaluation job, and a public code-execution agent should not inherit the same defaults merely because all three need a terminal.
Treat the container as one boundary, not the whole boundary
A standard container shares the node kernel. Security context, seccomp, capabilities, user namespaces, mandatory access controls, and Pod Security admission reduce risk, but the suitable combination depends on the threat model.
The Kubernetes Pod Security Standards define Baseline and Restricted profiles for common hardening requirements. Use Restricted as a starting point where the workload supports it: run as non-root, disallow privilege escalation, drop capabilities, and use an approved seccomp profile. Do not grant host namespaces, host paths, privileged mode, or broad device access to untrusted execution.
For a stronger boundary, consider a sandboxed container runtime or a virtual-machine-based runtime. Kubernetes RuntimeClass lets a workload select among runtime configurations made available by the cluster. The platform team must evaluate the runtime's isolation, compatibility, performance, accelerator support, observability, and operational maturity.
Node or cluster separation can be appropriate for high-risk workloads, regulated data, or incompatible runtime requirements. Scheduling isolation is useful only when admission policy prevents the workload from escaping the intended pool.
Give every execution a scoped identity
Create a dedicated ServiceAccount or other workload identity for the sandbox class. Disable automatic ServiceAccount token mounting when Kubernetes API access is unnecessary. Bind only the permissions needed by the controller or sidecar that manages the environment; the generated code often needs none.
Do not inject a broad credential and rely on the agent to use it responsibly. Prefer short-lived, task-scoped credentials issued after policy checks. If the agent acts on behalf of a user, preserve the subject, delegated scope, task, expiration, and approval context in the audit record.
Separate the identity that manages the sandbox from the identity available inside it. A control service may need permission to create and stop workloads without exposing that authority to the code being executed.
Make network access explicit
Start with no ingress and no egress, then allow the destinations required by the task. A typical execution may need DNS, one model endpoint, approved tool services, a package mirror, artifact storage, and telemetry collection. Each exception expands the sandbox's effective authority.
Kubernetes NetworkPolicy controls traffic where the network plugin enforces it. NetworkPolicy is based on selected Pods and allowed peers or IP blocks; it does not provide an application-aware allowlist for arbitrary hostnames. Address DNS behavior, proxies, changing provider addresses, and protocols that the network implementation may not cover.
Test the denied paths as well as the allowed ones. Attempt access to cluster metadata, the Kubernetes API, neighboring workloads, private address ranges, and unapproved public endpoints from the actual runtime. Record the policy and environment revisions with the result.
Separate scratch space from durable outputs
Give each execution a fresh writable workspace. Mount approved inputs read-only where practical, and avoid shared writable volumes between tenants. Set ephemeral-storage requests and limits, then enforce file count, archive expansion, and output-size constraints in the application when needed.
Decide which state is disposable, which state supports resume, and which evidence must be retained. A suspended agent workspace can preserve useful context, but it can also preserve compromised files. Resume only into a compatible security and application version, and reset the environment between independent tasks.
Export selected outputs through a controlled path. Do not automatically publish every file the agent created. Scan or validate artifacts before they become inputs to another workflow, an image build, or a production deployment.
Bound every scarce resource
Set CPU, memory, ephemeral storage, and execution deadlines. Apply process and file-descriptor limits through the runtime where appropriate. If the workload receives a GPU, understand which device interfaces and sharing mechanisms become visible; an accelerator request is also part of the security and tenancy contract.
Add application budgets for model calls, tokens, tool invocations, child tasks, and outbound data. Kubernetes resource limits cannot stop an agent from spending through a hosted API or creating many external side effects.
Cancellation should prevent new actions, revoke task credentials, and stop descendant work. Cleanup should remove workloads, temporary storage, network exceptions, leases, and warm-pool assignments without deleting the retained evidence needed for investigation.
Model the sandbox lifecycle explicitly
An agent workspace may be created, claimed, active, idle, suspended, resumed, expired, and destroyed. Represent those states rather than inferring them from whether a Pod exists.
The Kubernetes SIG Apps Agent Sandbox project is developing a Sandbox custom resource and extensions for stateful singleton workloads. Its public design includes stable identity, stronger runtime options such as gVisor or Kata Containers, scale-to-zero behavior, and warm pools that can satisfy claims from templates. The project is under active development; validate its current API, security properties, and release maturity before relying on it.
Even with a purpose-built controller, the application must define checkpoint consistency, credential lifetime, output handling, and tool authorization. Suspending compute does not automatically make external state durable or revoke authority.
Capture evidence at both layers
Retain Kubernetes audit events, admission decisions, workload status, runtime alerts, network flow evidence where available, and resource consumption. Connect those records to the agent task, execution attempt, user or tenant, and policy revision.
At the application layer, record model and tool activity with appropriate redaction, including the approved action, actual arguments, result or receipt, and evaluation outcome. A container trace can show that a process opened a connection; it cannot by itself explain whether the action was permitted for that task.
Use alerts for boundary violations and suspicious behavior, not only container failure. Examples include access to a denied destination, unexpected Kubernetes API calls, writes outside the workspace, repeated secret lookups, abnormal child-process creation, or continued activity after cancellation.
Configure the Polyaxon side accurately
Polyaxon sandboxes expose process, filesystem, and terminal access inside the main container of a running service. They use the service's image, user, mounts, connections, resources, and network configuration. Enabling the sandbox plugin does not create a separate workload kind or stronger isolation boundary.
Use workload security-context presets and the environment configuration supported by your deployment to apply container settings. Attach only the required connections and prefer managed, scoped credentials. Use explicit resource and timeout settings, and persist approved outputs through configured artifact storage.
For generated or hostile code, the cluster and runtime must supply the isolation required by the threat model. Keep the general AI sandbox guide as the definition of sandbox use cases; this article is the Kubernetes implementation checklist around that boundary.
Verify containment before production
Create synthetic credentials and data, then run a representative adversarial task. Confirm that it cannot read another workspace, call the Kubernetes API, reach denied destinations, retain files into an unrelated session, exceed its deadline, or publish unapproved outputs. Interrupt the node and controller to inspect cleanup and recovery behavior.
Repeat these checks after changes to the base image, runtime, network plugin, admission policy, credential broker, storage class, or sandbox controller. Isolation is a property of the whole deployed system, not the name of one resource.