Manage the lifecycle of a Polyaxon sandbox service

A Polyaxon sandbox session follows the lifecycle of its backing service run. You submit a component, Polyaxon schedules it on a compute agent, and the sandbox API becomes available inside the running container. The service remains available for repeated commands until it stops or fails.

That gives an ML team a common way to run an interactive investigation, connect an LLM to a Python environment, and preserve the results alongside other platform work. The key is to manage both the service run and the commands executing inside it.

Start with a service component

Complete the project and client setup in the Sandbox Quick Start, then clone the examples repository and open the shared lifecycle directory:

git clone https://github.com/polyaxon/polyaxon-examples.git
cd polyaxon-examples/blog/sandbox-lifecycle

Save this component as sandbox.yaml in that directory:

kind: component
version: 1.1
name: lifecycle-example

plugins:
  sandbox: true
  auth: false

termination:
  timeout: 1800

run:
  kind: service
  volumes:
    - name: workspace
      emptyDir: {}
  container:
    image: python:3.11
    workingDir: /workspace
    command: ["sleep", "infinity"]
    volumeMounts:
      - name: workspace
        mountPath: /workspace

The scratch volume and service command keep the example small. For team use, add reviewed resource settings, connections, and environment presets. Use an image containing the required dependencies, pinned to a digest when repeatability matters.

This example disables workload authentication because its command does not call the Polyaxon API. Host access still uses normal Polyaxon authentication and project permissions.

Wait for the run, then check the sandbox

Creation, approval, scheduling, and sandbox health are separate steps. The run client can submit a Polyaxonfile and wait for a lifecycle condition. The sandbox client then checks that process and file access are reachable.

The following host script assumes your configured identity may launch the service. Save it as run-lifecycle.py beside sandbox.yaml and the repository's lifecycle.py. Run it from that directory after completing the project and client setup. This shared helper sets a 10-second HTTP timeout with transport retries disabled, checks a 300-second readiness deadline between status calls, and saves a separate cleanup receipt. It uses the public run-client interfaces and your existing client configuration.

The script explicitly approves the submission, executes one command, saves a local receipt, and attempts to stop and confirm the service afterward:

import json
from pathlib import Path

from polyaxon.client import SandboxClient

from lifecycle import make_run_client, stop_and_record, wait_for_running

run_client = make_run_client(project="quick-start")
run = run_client.create_from_polyaxonfile(
    polyaxonfile="sandbox.yaml",
    approved=True,
)
print("Created run:", run.uuid)

try:
    wait_for_running(run_client, seconds=300)

    with SandboxClient(
        project="quick-start",
        run_uuid=run.uuid,
        client=run_client.client,
    ) as sandbox:
        sandbox.ping()
        result = sandbox.process.exec(
            command=["python", "-V"],
            timeout_ms=30_000,
        )
        receipt = {
            "run_uuid": run.uuid,
            "exit_code": result.exit_code,
            "timed_out": result.timed_out,
            "duration_ms": result.duration_ms,
            "stdout": result.stdout,
            "stderr": result.stderr,
        }
        Path("sandbox-receipt.json").write_text(
            json.dumps(receipt, indent=2),
            encoding="utf-8",
        )
        if result.timed_out or result.exit_code != 0:
            raise RuntimeError("Environment command failed")
finally:
    cleanup = stop_and_record(run_client, seconds=30)

if not cleanup["confirmed"]:
    raise RuntimeError("Command finished, but service termination needs review")

For a workflow with manual release approval, preserve that approval step before execution. SandboxClient.create() is another submission interface, but it does not approve a run or wait for it to become ready. Its returned UUID is the identity of the new service, not evidence that commands can already run.

A health check can still fail while the sandbox endpoint starts or because access or networking is misconfigured. Retain the UUID, inspect the run's status and logs, and use a bounded retry policy in unattended clients.

The polling deadline, HTTP timeout, command timeout, and service timeout cover different phases. The helper stops retrying status requests at its deadline; a request already in progress still follows the HTTP transport's timeout. It makes one cleanup attempt and polls for a terminal status, recording confirmed: false if access or timing prevents confirmation. An accepted stop request alone does not establish that the workload stopped. Use a writable receipt directory; if saving fails, the printed receipt includes receipt_storage_error. These examples have been reviewed against the client interfaces; they are not benchmarked timing guarantees.

Track command lifetime independently

One service can host multiple executions. Choose the process interface that matches the task:

A failing command does not necessarily stop the service. Closing SandboxClient releases host-side client resources; stop the run explicitly when the environment is no longer needed. A background handle's wait timeout is also distinct from the server-side execution limit.

For an agent application, keep the service run UUID and each background execution ID in host-owned task state. Reconnecting to the same service is useful for continuing an investigation; your application must decide whether the previous command finished before issuing it again.

Separate execution and resource deadlines

Polyaxon offers different controls for different durations:

• timeout_ms on a sandbox command bounds that execution in milliseconds.
• termination.timeout bounds the service operation in seconds.
• termination.culling stops an idle service based on an activity probe.
• termination.ttl controls retention of finished cluster resources for cleanup or inspection.

The termination reference documents culling and activity probes from v2.12. Configure a probe suited to the service; an agent's activity is not necessarily reflected by a notebook endpoint or HTTP request count.

Idle termination stops compute. The documented sandbox lifecycle does not provide an automatic snapshot-and-resume guarantee for in-memory processes. Build recovery around saved inputs, code, and artifacts.

Preserve outputs before stopping

The host receipt above is a local file. Likewise, uploading data to /workspace through sandbox.fs places it in the running container. Neither action registers an artifact in Polyaxon.

For durable platform results, a program with the tracking library and appropriate run authentication can write under tracking.get_outputs_path() and register artifact metadata. Follow Persist Outputs and confirm important results appear in the run's artifact view before cleanup. Downloads are useful for retrieving scratch files; component and artifact registries serve the reusable outputs.

Capture a discovered fix in Git, move dependencies into the image, and register the workflow as a component version. A future service can reconstruct the environment from those inputs, and a finite job can run the now-stable procedure without an interactive session.

Finish with an inspectable run record

Use the project dashboard to confirm the service has reached its final status and inspect retained logs and artifacts. A service timeout provides a useful backstop if the host process disappears before its cleanup code runs.

Stopping a service ends its workload, but separately provisioned database accounts, externally issued tokens, and application-side browser sessions follow their own lifetimes. Keep those resources in the application's cleanup procedure.

The lasting output of the session is the run identity, reviewed configuration, source changes, and useful artifacts. Those are what make the next investigation or automated execution easier to reproduce in Polyaxon.