Autoresearch as a controlled experiment

Autoresearch makes experiment selection programmable. A policy proposes a change, observes its result, and uses that observation to choose the next experiment. The resulting system searches over configurations, training procedures, or source code.

Karpathy's autoresearch provides a compact implementation: an agent modifies train.py, trains on a single GPU for a fixed five-minute budget, measures validation bits per byte, and retains useful changes. The editable file contains the model, optimizer, and training loop; prepare.py fixes the data and evaluation utilities.

Scaling this loop introduces a measurement problem and a systems problem. Results must remain comparable as proposals adapt to previous evaluations. Experiment identity and decisions must also survive failures, retries, and parallel execution.

This note describes Polyaxon's planned research campaign design. It connects a proposal policy to the existing execution platform through a durable controller and a separate Evals contract. The controller, native Evals product, and consistent agent CLI contract are roadmap work. The numerical example below is an analytical calculation, not a campaign benchmark.

The experiment protocol

A candidate's score is conditional on how it was measured. Comparing two training changes requires a fixed data revision, evaluator, environment and hardware, seed policy, and per-trial budget. These conditions form the experiment protocol; the candidate configuration or source artifact varies within it.

For example, a warmup change and a wider model can be compared under the same five-minute training limit. That comparison establishes performance within that limit on the specified hardware. It does not establish which candidate performs better after a longer training run or on another device.

Autoresearch's training budget excludes startup and compilation. A campaign must account for those costs as well as evaluation, unsuccessful runs, and proposal-model usage. A fixed training limit is one constraint inside the total research budget. Autoresearch instructions.

The campaign should record the protocol with every observation. Changing the evaluator, dataset, hardware, permitted edit scope, or seed policy creates a new campaign revision and requires an explicit decision about which earlier observations remain compatible. A judge update can change an apparent improvement without changing the candidate; our evaluator-revision example examines that case.

Reading campaign progress

The familiar autoresearch chart shows individual experiment results alongside a running-best step line. In the score-only convention shown here, retained improvements advance the frontier and discarded candidates remain visible above it. Failed executions belong in the history but have no score.

Constructed example history, with the baseline normalized to 1.00. The keep rule here is a strict score decrease; proposal labels are examples. Failed executions have no score and do not enter the frontier.

The points reveal the work behind the best result. A long plateau can contain many completed experiments and several failures. Each point should lead to its candidate artifact, protocol, execution evidence, and decision. The step line summarizes development selection; it does not independently confirm the selected candidate's quality.

Adaptive search changes evaluation

The frontier can move even when true quality does not. Evaluator noise produces favorable measurements, and repeatedly selecting against the same feedback makes the evaluation part of the search process. Adaptive reuse of evaluation data is a studied source of overfitting. Dwork et al., Generalization in Adaptive Data Analysis and Holdout Reuse.

A simple model isolates the selection effect. Suppose every candidate has the same true score as an exact, known reference, and each measured candidate score has independent Gaussian error with standard deviation σ. Take lower scores to be better. For a single candidate, the probability of an apparent improvement of at least k standard deviations is Φ(−k), where Φ is the standard normal cumulative distribution function. Across n candidates, the probability of observing at least one such improvement is:

Exact probabilities under the stated noise model. “2 SD” and “3 SD” are apparent gains of two and three standard deviations below the known reference. Candidate count is logarithmic.

At 100 candidates, the probability is approximately 90% for a two-standard-deviation apparent gain, even though no candidate is actually better. Real campaigns have correlated errors and history-dependent proposals, so this model does not estimate their error rate. It demonstrates why a favorable development frontier alone is insufficient evidence of improvement.

Our design separates development feedback, promotion checks, and final assessment. Development results can enter proposal context. Promotion access is limited and counted. Final holdout cases and results remain outside that context. These controls limit evaluation exposure; statistical validity still depends on the sampling, repetition, and decision procedure.

A campaign with separate proposal and execution authority

The proposed campaign has three responsibilities: generating candidates, enforcing the experiment protocol, and producing evaluation evidence. A model can participate in the first responsibility without acquiring authority over the other two.

Proposed development loop. The dashed boundary contains candidate execution. The evaluator runs outside it; promotion and final holdout evaluation are separate paths and are not returned through this feedback loop.

The proposal policy receives a fixed snapshot of selected compatible results, the experiment protocol, and remaining budgets. It emits a candidate tied to an immutable parent. Initially, candidates will be typed configuration changes. A numerical subproblem can be delegated to Hypertune over a declared parameter space; native TPE already supplies that form of search. LLM clients, source mutation, and campaign persistence remain outside the numerical optimizer.

The controller validates a proposal, obtains any required approval, and reserves budget before dispatching work. It owns the mapping from a proposal to its child run and the decision made from the resulting evidence. Replaying a completion event must recover the existing run and decision rather than create another experiment.

Evals will define the identities of the subject, data, evaluator, logical results, and physical executions. The campaign consumes that contract. Candidate-produced metrics are evidence to inspect; protected evaluation determines whether the result satisfies the campaign's decision rule.

Parallel proposals record their parents and retain distinct immutable candidate artifacts. Combining two successful changes creates a new candidate that needs its own evaluation. This preserves the distinction between evidence for each change and evidence for their interaction.

Polyaxon's planned campaign interface will expose proposals, lineage, decisions, and remaining budgets through existing run surfaces. Enforcement stays in the controller. The agent CLI work supplies stable structured responses, bounded queries, and error semantics so agents can drive that controller reliably.

One result, multiple executions

An infrastructure retry and an experimental repetition have different meanings. Consider one candidate, one evaluation case, and one declared repetition. If its worker is lost and execution is retried with the locked inputs, the two physical executions still belong to one logical attempt.

Proposed Evals identity. Both executions retain evidence and consume resources; the selected execution contributes one logical result. A retry does not create an additional repetition.

The failed execution remains in the cost and failure history. It does not become a benchmark score of zero, and the retry does not increase the aggregation denominator. A valid model failure is a different outcome and is scored according to the evaluator's contract.

This identity also governs recovery. A controller that restarts between dispatch and result publication must reconcile the physical execution and canonical evidence before committing a logical result. Run status alone cannot resolve every interrupted handoff. The result record must identify which execution was selected, whether evaluation completed, and why a retry or termination occurred.

Evaluating the researcher

The measurement target is the independently confirmed quality of the final candidate at a declared total campaign budget. Comparing the lowest development score from each policy would mix search effectiveness with selection effects.

For configuration-only campaigns, applicable baselines include random search and TPE under the same parameter space and protocol. Source-editing policies require comparisons under the same allowed files, edits, and dependencies; a numerical optimizer is not a direct baseline for unrestricted code changes. Campaign repetitions should vary the declared seeds and initial conditions, and reports should show the distribution of final outcomes.

The first campaign controller will use a scripted proposal policy so replay, budget enforcement, and result identity can be assessed before introducing provider variability. Configuration-only LLM proposals follow; isolated source patches require a restricted execution profile and protected evaluation. The default execution mode requires approval.

Polyaxon's current runs and comparison, artifacts and lineage, and sandboxes provide execution and evidence primitives. Today's sandbox plugin exposes access inside a service container; the planned source-experiment restrictions are additional work. Persistent workspace lifecycle is covered in the sandbox roadmap.

No campaign benchmark results or advantage for LLM-generated proposals are reported here. The planned work makes those claims measurable: a declared protocol, a reproducible candidate, independent evaluation, and a complete accounting of the search that produced them. Availability will be tracked on the Polyaxon roadmap.