Polyaxon v3 is coming →

Budget scratch storage for AI agent sandboxes on Kubernetes

Size an agent sandbox's working directory, set ephemeral storage requests and limits, and save the files that should survive the Pod.

November 22, 2026by Polyaxon

A coding agent downloads a 600 MiB archive, extracts a 2 GiB repository, then writes build output while the original archive is still on disk. Its result is a small patch. The temporary files decide whether the sandbox stays alive long enough to save it.

Give each sandbox a disk budget based on what coexists at the busiest step. In Kubernetes, emptyDir.sizeLimit, an ephemeral-storage request, and an ephemeral-storage limit each control a different part of that budget.

Kubernetes logo above blue waves.

Work out what shares the disk

For an illustrative single-container sandbox, suppose the archive, extracted repository, and build files peak together near 3 GiB. Allow another 1 GiB in the expected working set for logs and files written outside /scratch. Start with these settings and revise them after observing real workloads.

SettingExampleDecision it controls
Container ephemeral-storage request4GiThe scheduler looks for a node that can fit the requested local storage.
Disk-backed /scratch emptyDir.sizeLimit4GiThe scratch volume has its own usage threshold. It can fill sooner if the node runs short of space.
Container ephemeral-storage limit6GiIn this single-container Pod, the total local budget covers the scratch volume, writable layer, and logs.

The volume budget fits inside the Pod's budget. In this example, the kubelet can start eviction when tracked Pod usage exceeds 6 GiB or when /scratch exceeds its own 4 GiB threshold. In a multi-container Pod, include the limits of injected sidecars when working out the aggregate limit, and account for their logs and writable layers. A container can also trigger eviction by exceeding its own limit with its logs and writable layer. Kubernetes local ephemeral storage

Use the application to reject a download that would overrun the remaining allowance, including its expected unpacked size. Put an upper bound on extraction and build output as well. The kubelet checks usage and can evict the Pod after a limit is exceeded. Treat its thresholds as a backstop for keeping node capacity available. Kubernetes emptyDir volumes

Give the sandbox a known scratch directory

For a Polyaxon sandbox, merge this fragment into an existing working sandbox Polyaxonfile. It assumes Polyaxon 2.18 or later. Keep your image, service configuration, existing mounts, and other resource requests. The native Kubernetes resource keys go under container.resources. The root volumes shortcut adds the disk-backed volume to the workload Pod. If your resource settings already use a root resources block, consolidate them under container.resources for this example. Resource configuration · Polyaxon 2.18 execution shortcuts

volumes:
  - name: agent-scratch
    emptyDir:
      sizeLimit: 4Gi
container:
  volumeMounts:
    - name: agent-scratch
      mountPath: /scratch
  resources:
    requests:
      ephemeral-storage: 4Gi
    limits:
      ephemeral-storage: 6Gi
env:
  TMPDIR: /scratch

Configure checkout, extraction, and build directories explicitly. TMPDIR only affects tools that honor it. Make sure the sandbox user can write to the mount. Use disk backing here, since a memory-backed emptyDir consumes memory instead. emptyDir survives a container restart in the same Pod. Kubernetes removes it when the Pod is removed from its node. Kubernetes emptyDir lifecycle

Decide which files are worth keeping

Keep downloads, extracted inputs, and intermediate builds on scratch. Save the patch, result manifest, and relevant logs as soon as the useful step completes. In Polyaxon, write them to the run's configured outputs path and confirm that they reached the artifact store, or download the selected files from the running sandbox. A link to the artifact in tracking metadata cannot recover a file that only existed on a deleted Pod. Persist sandbox outputs

Use separate storage for large shared inputs if many sandboxes use them. A prepared read-only connection can supply pinned data or weights, leaving scratch for changes specific to one task. Only assign a durable writable workspace when users actually need to resume files between sessions. Workspace state lifetimes · Read-only cache walkthrough

Use the failure reason to decide what to change

What you observeNext check
The new sandbox stays Pending with an insufficient ephemeral storage eventCompare the Pod's requests with allocatable storage and existing reservations on eligible nodes.
/scratch grows while Pod logs stay smallMeasure the archive, extraction, and build together. Remove intermediate files when each phase completes.
A Pod is evicted and its scratch volume stayed below 4 GiBCheck writable layers, container logs, and the eviction message. The aggregate budget can be exceeded elsewhere.
Several Pods are evicted as the node reports DiskPressureInspect free bytes and inodes on the affected filesystem. Image layers and other workloads can exhaust space even when this sandbox stays within budget.

Start with the Pod's events and the node condition. Some node layouts limit what kubelet storage accounting can see. If usage and eviction evidence disagree, check the cluster's filesystem configuration before raising the budget. Kubernetes node-pressure eviction