Let an agent drive ML experiments on Polyaxon

Run the agent itself as a Polyaxon job or service and let it coordinate the ML work. Given a task such as improving a classifier's preprocessing, the agent can inspect data in a sandbox, launch training and evaluation jobs, read their results, and decide which experiment to try next.

The controller and the workloads it launches each have their own execution records, resources, and lifetime. Polyaxon schedules and tracks them; the agent application implements the decision loop and its permitted tools. Application-owned tools can run inside the platform alongside the agent.

The platform-integration guide covers tool contracts and shared execution records. This article follows the hosted controller's experiment loop.

The loop runs in the agent application hosted on Polyaxon. Submission, command execution, logging, and version registration remain separate actions.

Run the agent controller on Polyaxon

Use a job for a bounded agent task or a service for an agent that receives ongoing requests. Package the controller, its framework, and the Polyaxon client in the runtime image. It can call an external LLM API or a model-serving endpoint on Polyaxon.

With runtime authentication enabled, in-cluster clients can load the project/run context and scoped API credentials. The controller needs permission to create and inspect the workloads it manages; running inside the platform does not grant additional access by itself.

For the preprocessing task, the controller can:

1. Open or reuse a sandbox to inspect a data sample and try a code change.
2. Commit the selected change and resolve the image and dataset versions for a trial.
3. Submit reviewed training and evaluation components on the allowed compute queues.
4. Inspect their status, metrics, and artifacts, then compare the candidate with the baseline.
5. Choose another permitted trial, stop at the task's budget, or register a candidate when the acceptance policy permits it.

The controller uses two execution paths:

• A job for a defined training, analysis, or evaluation task. Submit a reviewed component through RunClient and retain its run UUID.
• A sandbox-enabled service for repeated commands and workspace files. Use SandboxClient for process and file access inside that running service.

The sandbox supports exploration; the versioned job supports comparison and reuse. Temporary package installs and edited files in a live session are not enough to reproduce the later job.

A sandbox service has its own run UUID. Background commands started with sandbox.process.exec_bg() also return execution IDs; use those to inspect their output or status while the service is available. A command can fail while the service remains running. The sandbox automation guide shows these interfaces.

Follow each proposal to its executions

Associate the application request, agent step, Polyaxon execution, and output artifacts when the work is created. Different identifiers answer different questions:

The controller keeps the proposal-to-execution mapping, including its own run UUID and the UUIDs it creates. Log the proposed change and the resulting decision against the controller's run. Record input/output artifact references on the ML runs so the evidence can be inspected through run lineage.

Retain the agent model, prompt revision, and tool version with the controller's task context. These describe how the agent chose an action; the training job's code, image, and data describe what actually ran. Use tracking metadata for compact references.

Allocate compute per role

When model inference runs separately, the controller can use CPU resources while its training jobs use GPU queues. Each trial requests its own compute, so the controller's resource allocation does not determine the size of the training job.

Use resource requests and limits to define the CPU, memory, and GPU requirements of the reviewed workload. Where commercial queues are enabled, map the tool to an approved queue on the compute agent managing the target Kubernetes cluster or namespace. Queues add routing, priorities, concurrency, and quota controls; the job can still wait for capacity.

Connections are configured per workload. The controller needs its model/API access, a sandbox may need a data sample, and training may need the full dataset, model cache, and output storage. Keep those requirements with the corresponding workload definitions.

For commercial deployments, give the controller a scoped Polyaxon service account for its API calls. Runs and activity can then be attributed to that identity rather than a researcher's personal token.

Preserve the controller's progress

Keep submitted run identifiers and the current task step in durable controller state. After a controller restart, inspect the existing work before deciding whether to submit again. The training run's lifetime can extend beyond the controller process that launched it. Define whether outstanding work should finish or be stopped when the agent task is cancelled.

For substantial jobs, inspect progress separately from submission. A queued job, a running sandbox service, and a completed command are different states; check the state relevant to the requested action before choosing the next step.

Persist useful files before stopping or replacing a sandbox. Write retained outputs into the run's synced outputs path and log their artifact references, or explicitly download scratch files. log_artifact_ref() records metadata; it does not copy an arbitrary workspace file into durable storage. The output-persistence guide shows both paths.

Use termination settings to bound workload duration. A command timeout and a service's lifetime are separate controls. Keep request-level budgets and retry decisions in the application; for duplicate submissions and uncertain responses, see the worked MCP adapter.

Choose the next experiment

For the preprocessing change, compare the candidate and baseline using the same evaluation data and settings. A job completing successfully establishes execution success; its metrics can still show that the candidate is worse.

The controller uses that result to decide whether to retain the change, try another permitted proposal, or stop. Treat an execution failure as a failure to obtain a comparable result, rather than assigning it a model-quality score. Stop when the task's budget or completion condition is reached, and retain the decisions with their run references.

Register a candidate in the model registry when it needs a managed version, retaining its source-run and artifact lineage. Registration, promotion, and deployment are explicit actions governed by the team's acceptance policy. The controller can invoke those actions when its policy allows them, with the relevant runs and evidence available for review.