Observe coding agents from prompt to sandbox execution

A coding agent that fixes a training script can leave several different records: a model response in the agent application, a command result in a sandbox, a patch in a repository, and an evaluation report in another process. Investigating a regression means connecting those records to the exact environment and candidate that produced them.

Polyaxon gives this workflow a common foundation. A sandbox is a service run with an image, configuration, status, and run UUID. Evaluation jobs have the same tracking and artifact model. By recording the relationship between the agent task, sandbox run, and qualification run, a platform team can move from an unsuccessful task to its execution evidence and then compare a proposed fix with a baseline.

Choose the run and task boundaries

Use a task identifier for one requested change, such as repairing an ML preprocessing function. Record the repository revision, prompt revision, model settings, tool definitions, and evaluation manifest before the first attempt.

A sandbox run may serve several steps of that task. Individual calls to the sandbox process API execute inside its main container; they do not create a new Polyaxon run for each command. Keep a task ID and step number in the evidence your application records so retries remain distinguishable.

Use a separate evaluation job when the accepted patch needs an independent environment, a repeatable case suite, or its own scheduling and resource settings. Record the sandbox UUID and patch digest as evaluation inputs. This makes the platform relationship explicit without assuming that the SDK automatically links every application event.

Record a command receipt in a tracked operation

The following example runs in a trusted Polyaxon job or service whose image contains the Polyaxon client. Its tracking context and artifact collection are enabled. Set SANDBOX_OWNER, SANDBOX_PROJECT, SANDBOX_RUN_UUID, and AGENT_TASK_ID in that trusted application. The target must already be a running sandbox-enabled service with Python available.

This small calculation demonstrates recording a receipt without retaining arbitrary stdout. Replace the fixed command with your application's approved tool action after defining its authorization policy.

import json
import os
from pathlib import Path

from polyaxon import tracking
from polyaxon.client import SandboxClient
from polyaxon.schemas import V1ArtifactKind

tracking.init()
sandbox_uuid = os.environ["SANDBOX_RUN_UUID"]
task_id = os.environ["AGENT_TASK_ID"]
tracking.log_inputs(task_id=task_id, sandbox_run_uuid=sandbox_uuid)

with SandboxClient(
    owner=os.environ["SANDBOX_OWNER"],
    project=os.environ["SANDBOX_PROJECT"],
    run_uuid=sandbox_uuid,
) as sandbox:
    sandbox.ping()
    result = sandbox.process.exec(
        command=["python", "-c", "print(sum([2, 3, 5]))"],
        timeout_ms=10_000,
    )

receipt = {
    "task_id": task_id,
    "sandbox_run_uuid": sandbox_uuid,
    "step": 1,
    "tool": "calculate_fixture_total",
    "exit_code": result.exit_code,
    "duration_ms": result.duration_ms,
    "timed_out": result.timed_out,
    "stdout_truncated": result.stdout_truncated,
    "stderr_truncated": result.stderr_truncated,
    "fixture_matches": result.stdout.strip() == "10",
}
path = tracking.get_outputs_path("receipts/step-1.json")
Path(path).write_text(json.dumps(receipt, indent=2), encoding="utf-8")
tracking.log_artifact_ref(
    path=path,
    kind=V1ArtifactKind.FILE,
    name="command-receipt",
)
tracking.log_metrics(
    step=1,
    command_duration_ms=result.duration_ms,
    command_timed_out=int(result.timed_out),
    fixture_matches=int(receipt["fixture_matches"]),
)

The receipt belongs to the trusted operation running this script, and explicitly identifies the sandbox it contacted. get_outputs_path() places the file in that operation's outputs directory. log_artifact_ref() registers its lineage metadata; it does not copy a file from an arbitrary sandbox path. The artifact tracking guide explains the distinction.

Separate operational completion from task acceptance

A zero exit code means the command completed successfully according to the process. It does not establish that a patch fixed the reported defect, preserved an API contract, or avoided unrelated changes. Log these as separate results.

For each candidate, retain a case report with expected behavior, observed behavior, evaluator revision, and acceptance decision. Use tracking inputs and outputs for stable revisions and final decisions, and metrics for measurements you want to chart.

Keep latency categories distinct. Command duration_ms measures sandbox execution; it does not include queueing, sandbox provisioning, model requests, or human review. Capture those durations in the host application or qualification workflow. Likewise, calculate model usage and cost from actual provider responses and your price basis before logging them. Sandbox runtime is not a measure of token spend.

Investigate through Polyaxon views

Start from the failed candidate's run and inspect its inputs: prompt, tool, source, and case-set revisions. Open the receipt and patch artifacts to determine whether the failure came from an invalid proposal, a timed-out command, truncated evidence, or a functionally incorrect change.

Then inspect the run's resource monitoring and logs. A memory-constrained process needs a different response from an agent repeatedly choosing the wrong tool. For a live sandbox, debug the session while preserving the files needed for reproduction.

In run comparison, filter candidates to the same case-set revision and display acceptance, timeout rate, command duration, and application-recorded cost. Sort by the metric that failed its requirement, then inspect case artifacts rather than selecting solely on the average.

Feed the evidence into the next release

Convert a sanitized failure into a fixed evaluation case and rerun it through a versioned qualification component. Preserve the initial repository state and expected patch behavior. Compare the repaired agent with its baseline across both the failing case and the broader suite.

The useful result is a reviewable chain in Polyaxon: candidate configuration, sandbox receipt, generated artifact, evaluation evidence, and release decision. That chain makes coding-agent observability part of the platform's normal experimentation and release workflow.