Optimize a small-model harness with autoresearch on Polyaxon

A pronunciation classifier can return a valid answer and still solve the wrong problem. In 银行, 行 belongs to the word for a bank. In 步行, it belongs to the word for walking. Asking about the character's grammatical role without identifying the containing word leaves the model to recover information the application could have supplied.

This guide develops a Polyaxon workflow for optimizing the code around a fixed model: propose a change, probe its mechanism, evaluate the candidate, inspect regressions, and retain the evidence behind the decision.

The implementation uses current jobs, services, sandboxes, matrices, tracking, and artifacts. Your application owns the research policy and acceptance rule. The native campaign controller and Evals product described in the autoresearch roadmap remain planned capabilities. This is a runnable experiment design, not a report of measured Polyaxon quality or speed improvements.

Define the decision and its denominator

The task is to choose a reading for one marked character occurrence. The model receives the sentence, the character's position, and a fixed set of allowed readings. It never receives the expected answer.

Each case has a stable ID, document group, explicit offset, allowed readings, and reviewed label. Save this teaching manifest as cases.json:

[
  {
    "case_id": "teaching-bank-01", "document_id": "teaching-bank",
    "sentence": "银行今天营业。", "target_index": 1, "character": "行",
    "criteria": {
      "hang2": "A business, profession, institution, or row.",
      "xing2": "Movement, travel, conduct, or being acceptable."
    },
    "expected": "hang2"
  },
  {
    "case_id": "teaching-walk-01", "document_id": "teaching-walk",
    "sentence": "他每天步行上班。", "target_index": 4, "character": "行",
    "criteria": {
      "hang2": "A business, profession, institution, or row.",
      "xing2": "Movement, travel, conduct, or being acceptable."
    },
    "expected": "xing2"
  },
  {
    "case_id": "teaching-trade-01", "document_id": "teaching-trade",
    "sentence": "这个行业正在变化。", "target_index": 2, "character": "行",
    "criteria": {
      "hang2": "A business, profession, institution, or row.",
      "xing2": "Movement, travel, conduct, or being acceptable."
    },
    "expected": "hang2"
  }
]

Offsets refer to Python Unicode code points in the stored sentence. Normalizing or inserting marker characters after assigning offsets changes the task. Validate that the character at target_index matches character before making requests.

These three sentences demonstrate the input contract. They are not a benchmark or enough evidence for a model comparison.

On a real corpus, report errors on positions that require a pronunciation decision, with the number of such positions beside the count. Also report errors by character and reading, including uncommon readings. A majority-reading baseline can look strong on aggregate while failing the cases the product needs help with.

Split by document before extracting character cases, so another sentence or occurrence from the same document cannot leak across splits. Reserve development documents, promotion documents, and a final assessment set. To evaluate transfer, reserve character types that the policy never optimizes as well. A whole-corpus pass containing development cases is a regression audit, not a held-out generalization result.

Fix the model and vary its harness

There are two practical classifier backends:

TypeSafe's Choice API accepts state and typed questions. Our Qwen classifier deployment guide supplies the self-hosted alternative. The implementation below uses the common Choice request subset documented by both backends; their other features and probability semantics are not interchangeable. Simple Jev's selected-token probabilities are uncalibrated scores, so a large value does not establish that an answer is correct.

Configure a Polyaxon connection named decision-endpoint with CLASSIFIER_ENDPOINT, CLASSIFIER_MODEL, EXPECTED_MODEL_ID, CLASSIFIER_AUTH_SCHEME, and CLASSIFIER_TOKEN. Use Bearer for hosted Jev or token for a Polyaxon service. Preserve the full service URL prefix. Resolve the expected model identity before the study; the evaluator rejects responses from another model instead of silently comparing a moving alias.

Keep the model, dataset, evaluator, normalization, dependencies, request limits, and resource profile fixed within a comparison. Candidate requests, evidence policies, and routing rules are the variables. A different checkpoint or evaluator starts a different comparison.

Compute evidence before asking for a reading

A lexical candidate must cover the target occurrence, not merely appear somewhere in the sentence. Build candidates by joining character offsets to word spans. For each candidate, retain the word, the target syllable's reading, its gloss, and whether the segmenter produced exactly that span.

jieba supplies token positions. A production lexicon can come from a reviewed CC-CEDICT snapshot, with its version and license retained. For the teaching cases, a small hand-authored lexicon.json is sufficient:

[
  {"word": "银行", "readings": ["yin2", "hang2"], "gloss": "bank"},
  {"word": "步行", "readings": ["bu4", "xing2"], "gloss": "to walk"},
  {"word": "行业", "readings": ["hang2", "ye4"], "gloss": "industry or trade"}
]

Convert dictionary data into the same tone representation as the labels before running the study: lowercase numbered syllables, v for ü, and 5 for the neutral tone. Decide whether the target is lexical pronunciation or surface tone sandhi. Do not erase a tone disagreement just to make two sources agree.

The code below tests five harness modes plus a phrase-table baseline:

Keep the phrase-table baseline in the study. Where deterministic lookup already settles the case, a model can add cost without improving the answer. The classifier path needs evidence of value on the remaining ambiguous or unresolved cases.

Segmentation agreement is a feature to evaluate, not an authority over the label. The boundary mode can lose useful evidence when the segmenter is wrong. Compare it with lexicon rather than assuming filtering must help.

The second model call also needs a measured justification. A candidate can apply chain only to one character while leaving the others on the definitions baseline. For cases with no eligible word evidence it makes a single reading request. Choosing which characters receive a second call is itself a development decision.

Make each candidate a tracked evaluation job

Put the cases in cases.json, a JSON array, and the lexicon in lexicon.json. Keep the development files in a configured read-only pronunciation-dev connection mounted at /data/dev. Promotion data belongs to a separate evaluator connection and is absent from the research workspace.

Save the following as evaluate.py. It implements the six modes, bounded HTTP calls, repetitions, per-case evidence, and summary outputs. The proposer can change a small JSON configuration, including reading descriptions for one character; it cannot change the scoring function through that configuration.

import argparse
import hashlib
import json
import math
import os
import time
from importlib.metadata import version
from pathlib import Path
from statistics import median

import jieba
import requests
from pypinyin import Style, lazy_pinyin
from polyaxon import tracking
from polyaxon.schemas import V1ArtifactKind


def digest(path):
    return hashlib.sha256(Path(path).read_bytes()).hexdigest()


def prepare_lexicon(entries):
    by_word = {}
    for entry in entries:
        word = entry["word"]
        if not word or len(word) != len(entry["readings"]):
            raise ValueError("Lexicon syllables must align with word characters")
        by_word.setdefault(word, []).append(entry)
    return by_word, max((len(word) for word in by_word), default=0)


def word_evidence(case, lexicon_index, matched_only, limit):
    sentence, target = case["sentence"], case["target_index"]
    spans = {(start, end) for _, start, end in jieba.tokenize(sentence)}
    by_word, longest = lexicon_index
    evidence = []
    for start in range(max(0, target - longest + 1), target + 1):
        for end in range(target + 1, min(len(sentence), start + longest) + 1):
            word = sentence[start:end]
            for entry in by_word.get(word, []):
                reading = entry["readings"][target - start]
                matched = (start, end) in spans
                if reading not in case["criteria"] or (matched_only and not matched):
                    continue
                evidence.append({"word": word, "start": start, "end": end,
                                 "reading": reading, "gloss": entry["gloss"],
                                 "token_span_match": matched})
    evidence.sort(key=lambda item: (not item["token_span_match"],
                                   -(item["end"] - item["start"]),
                                   item["start"], item["word"], item["reading"]))
    return evidence[:limit]


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--cases", required=True)
    parser.add_argument("--lexicon", required=True)
    parser.add_argument("--proposal", required=True)
    parser.add_argument("--repeats", type=int, default=3)
    parser.add_argument("--max-calls", type=int, default=2000)
    args = parser.parse_args()
    cases = json.loads(Path(args.cases).read_text(encoding="utf-8"))
    lexicon = json.loads(Path(args.lexicon).read_text(encoding="utf-8"))
    lexicon_index = prepare_lexicon(lexicon)
    proposal = json.loads(args.proposal)
    allowed = {"mode", "target", "limit", "criteria", "routes"}
    modes = {"phrase", "labels", "definitions", "lexicon", "boundary", "chain"}
    if not isinstance(proposal, dict) or set(proposal) - allowed:
        raise ValueError("Unexpected proposal fields")
    mode, target = proposal.get("mode"), proposal.get("target", "all")
    limit, overrides = proposal.get("limit", 4), proposal.get("criteria", {})
    if not isinstance(mode, str) or mode not in modes or type(limit) is not int or not 1 <= limit <= 4:
        raise ValueError("Invalid mode or evidence limit")
    if type(args.repeats) is not int or args.repeats < 1 or args.max_calls < 0:
        raise ValueError("Invalid repetition or request budget")
    if not isinstance(overrides, dict) or any(
        not isinstance(value, str) or len(value) > 1000 for value in overrides.values()
    ):
        raise ValueError("Invalid reading descriptions")
    if overrides and target == "all":
        raise ValueError("Description edits must target one character")
    if not cases or len({case["case_id"] for case in cases}) != len(cases):
        raise ValueError("Cases must be nonempty and have unique IDs")
    if not isinstance(target, str) or (target != "all" and len(target) != 1):
        raise ValueError("Target must be all or one character")
    routes = proposal.get("routes", {})
    if not isinstance(routes, dict) or any(
        not isinstance(key, str) or len(key) != 1 for key in routes
    ):
        raise ValueError("Route keys must be single characters")
    for variant in routes.values():
        if not isinstance(variant, dict) or set(variant) - {"mode", "limit", "criteria"}:
            raise ValueError("Unexpected route fields")
        if (not isinstance(variant.get("mode"), str) or variant.get("mode") not in modes
                or type(variant.get("limit", 4)) is not int
                or not 1 <= variant.get("limit", 4) <= 4):
            raise ValueError("Invalid route mode or evidence limit")
        descriptions = variant.get("criteria", {})
        if not isinstance(descriptions, dict) or any(
            not isinstance(value, str) or len(value) > 1000 for value in descriptions.values()
        ):
            raise ValueError("Invalid route descriptions")
    for case in cases:
        index = case["target_index"]
        if (type(index) is not int or not 0 <= index < len(case["sentence"])
                or case["sentence"][index] != case["character"]
                or case["expected"] not in case["criteria"]
                or len(case["criteria"]) < 2):
            raise ValueError("Invalid case contract")
        if case["character"] == target and set(overrides) - set(case["criteria"]):
            raise ValueError("Descriptions cannot introduce new reading IDs")
        routed = routes.get(case["character"], {}).get("criteria", {})
        if set(routed) - set(case["criteria"]):
            raise ValueError("Route descriptions cannot introduce new reading IDs")
    def case_config(case):
        fallback = proposal if target == "all" or case["character"] == target else {"mode": "definitions"}
        return routes.get(case["character"], fallback)

    uses_model = any(case_config(case)["mode"] != "phrase" for case in cases)
    if uses_model:
        for key in ["CLASSIFIER_ENDPOINT", "CLASSIFIER_MODEL", "EXPECTED_MODEL_ID",
                    "CLASSIFIER_AUTH_SCHEME", "CLASSIFIER_TOKEN"]:
            if not os.environ.get(key):
                raise ValueError("Missing classifier configuration: " + key)

    tracking.init()
    tracking.log_inputs(proposal=proposal, cases_sha256=digest(args.cases),
                        lexicon_sha256=digest(args.lexicon),
                        harness_sha256=digest(__file__), repeats=args.repeats,
                        max_calls=args.max_calls,
                        requested_model=os.environ.get("CLASSIFIER_MODEL") if uses_model else "phrase",
                        expected_model=os.environ.get("EXPECTED_MODEL_ID") if uses_model else "phrase")
    root = Path(tracking.get_outputs_path("evaluation", is_dir=True))
    root.mkdir(parents=True, exist_ok=True)
    attempts, outcomes = [], []
    session = requests.Session()

    def choose(state, instructions, criteria):
        if len(attempts) >= args.max_calls:
            raise RuntimeError("Request budget exhausted")
        attempt = {"case_id": current["case_id"], "repetition": repetition,
                   "stage": instructions, "status": "submitted"}
        attempts.append(attempt)
        started = time.perf_counter()
        try:
            response = session.post(
                os.environ["CLASSIFIER_ENDPOINT"],
                headers={"Authorization": os.environ["CLASSIFIER_AUTH_SCHEME"]
                         + " " + os.environ["CLASSIFIER_TOKEN"]},
                json={"model": os.environ["CLASSIFIER_MODEL"], "state": state,
                      "questions": {"decision": {"type": "choice",
                                      "instructions": instructions,
                                      "criteria": criteria}}},
                timeout=(5, 25),
            )
            attempt["http_status"] = response.status_code
            response.raise_for_status()
            body = response.json()
            if not isinstance(body, dict):
                raise ValueError("Classifier response must be an object")
            attempt.update(model=body.get("model"), usage=body.get("usage"),
                           metadata=body.get("metadata"), status="returned")
            if body.get("model") != os.environ["EXPECTED_MODEL_ID"]:
                raise RuntimeError("Model identity differs from the protocol")
            answers = body.get("answers")
            answer = answers.get("decision", {}) if isinstance(answers, dict) else {}
            attempt["answer"] = answer
            choice = answer.get("choice") if isinstance(answer, dict) else None
            return choice if isinstance(choice, str) and choice in criteria else None
        except (requests.RequestException, ValueError, RuntimeError) as exc:
            attempt["status"] = "execution_error"
            attempt["error_type"] = type(exc).__name__
            raise
        finally:
            attempt["latency_ms"] = (time.perf_counter() - started) * 1000

    for repetition in range(args.repeats):
        for current in cases:
            started = time.perf_counter()
            row = {"case_id": current["case_id"],
                   "document_id": current["document_id"],
                   "character": current["character"], "expected": current["expected"],
                   "repetition": repetition, "prediction": None}
            config = case_config(current)
            active = config["mode"]
            try:
                if active == "phrase":
                    readings = lazy_pinyin(current["sentence"], style=Style.TONE3,
                                          neutral_tone_with_five=True,
                                          errors=lambda text: list(text))
                    if len(readings) != len(current["sentence"]):
                        raise RuntimeError("Phrase baseline lost character alignment")
                    row["prediction"] = readings[current["target_index"]]
                else:
                    criteria = dict(current["criteria"])
                    if active == "labels":
                        criteria = {key: key for key in criteria}
                    else:
                        criteria.update(config.get("criteria", {}))
                    state = {"sentence": current["sentence"],
                             "target_index": current["target_index"],
                             "target_character": current["character"]}
                    evidence = []
                    if active in {"lexicon", "boundary", "chain"}:
                        evidence = word_evidence(current, lexicon_index,
                                                 active != "lexicon", config.get("limit", 4))
                        state["word_evidence"] = evidence
                    row["evidence"] = evidence
                    valid_word_choice = True
                    if active == "chain" and evidence:
                        words = {"w" + str(i): item for i, item in enumerate(evidence)}
                        word_options = {key: json.dumps(value, ensure_ascii=False)
                                        for key, value in words.items()}
                        word_options["standalone"] = "No listed word contains this occurrence."
                        selected = choose(state, "Which word contains the target occurrence?",
                                          word_options)
                        valid_word_choice = selected is not None
                        state["selected_word"] = words.get(selected)
                    if valid_word_choice:
                        row["prediction"] = choose(
                            state, "Choose the pronunciation of the target occurrence.", criteria)
                row["status"] = ("ok" if row["prediction"] in current["criteria"] else "invalid")
            except (requests.RequestException, ValueError, RuntimeError) as exc:
                row.update(status="execution_error", error_type=type(exc).__name__)
            row["latency_ms"] = (time.perf_counter() - started) * 1000
            row["correct"] = row["status"] == "ok" and row["prediction"] == row["expected"]
            outcomes.append(row)

    session.close()
    completed = [row for row in outcomes if row["status"] != "execution_error"]
    latencies = sorted(row["latency_ms"] for row in completed)
    metrics = {"positions": len(cases), "repetitions": args.repeats,
               "requested": len(outcomes), "completed": len(completed),
               "incomplete": len(outcomes) - len(completed),
               "errors": sum(not row["correct"] for row in completed),
               "invalid": sum(row["status"] == "invalid" for row in completed),
               "request_attempts": len(attempts)}
    if latencies:
        metrics.update(p50_ms=median(latencies),
                       p95_ms=latencies[math.ceil(.95 * len(latencies)) - 1])
    for key in ["input_tokens", "output_tokens"]:
        values = [(attempt["usage"].get(key)
                   if isinstance(attempt.get("usage"), dict) else None)
                  for attempt in attempts]
        if values and all(type(value) is int and value >= 0 for value in values):
            metrics[key] = sum(values)
    files = {"cases.json": outcomes, "requests.json": attempts,
             "case-manifest.json": cases, "lexicon.json": lexicon,
             "proposal.json": proposal, "summary.json": metrics,
             "packages.json": {name: version(name)
                               for name in ["polyaxon", "requests", "jieba", "pypinyin"]}}
    for name, value in files.items():
        path = root / name
        path.write_text(json.dumps(value, ensure_ascii=False, indent=2), encoding="utf-8")
        tracking.log_artifact_ref(path=str(path), name=name, kind=V1ArtifactKind.FILE)
    tracking.log_metrics(**metrics)
    tracking.log_outputs(**metrics)
    if metrics["incomplete"]:
        raise RuntimeError("Incomplete evaluation; retained evidence, excluded from selection")


if __name__ == "__main__":
    main()

The lexicon is indexed once; each lookup enumerates spans around the target instead of scanning every dictionary entry. The pypinyin baseline checks output alignment before indexing a reading. The model request contains an explicit allowlist of case fields; expected and document labels never enter state. Every submitted request consumes the job's call budget, including requests that time out. There are no automatic HTTP retries.

An invalid returned choice counts as an incorrect result. An unresolved execution failure makes the evaluation incomplete and the job fails after retaining its evidence. Do not rank that job's partial error count against a complete candidate. A later infrastructure retry is another physical execution of the same intended evaluation, not an extra independent case.

The summary counts completed case attempts across repetitions. positions records the number of unique cases; repetitions records how many passes were requested. Use the case artifact for comparisons within each repetition rather than treating those repeated calls as independent positions.

Build one evaluator image containing this file and the four packages, then record its digest. For example:

FROM python:3.12-slim
WORKDIR /app
RUN python -m pip install --no-cache-dir polyaxon requests jieba pypinyin \
    && python -m pip freeze > /app/resolved-requirements.txt
COPY evaluate.py /app/evaluate.py

Build once for a study. Retain the resolved dependency list and use the image digest in subsequent jobs; rebuilding from these unpinned install instructions can change the environment.

Save this component as evaluate.yaml, replacing the image reference and configuring the two named connections:

version: 1.1
kind: component
name: pronunciation-eval
inputs:
  - name: proposal
    type: str
    value: '{"mode":"definitions","target":"all","limit":4,"criteria":{}}'
  - name: repeats
    type: int
    value: 3
  - name: max_calls
    type: int
    value: 2000
cache:
  disable: true
termination:
  timeout: 1800
  maxRetries: 0
run:
  kind: job
  connections: [pronunciation-dev, decision-endpoint]
  container:
    image: YOUR_REGISTRY/pronunciation-eval@sha256:YOUR_IMAGE_DIGEST
    command: [python, /app/evaluate.py]
    args:
      - --cases
      - /data/dev/cases.json
      - --lexicon
      - /data/dev/lexicon.json
      - --proposal
      - "{{ proposal }}"
      - --repeats
      - "{{ repeats }}"
      - --max-calls
      - "{{ max_calls }}"
    resources:
      requests:
        cpu: "1"
        memory: "2Gi"
      limits:
        cpu: "1"
        memory: "2Gi"

The CPU and memory values are starting allocations for the evaluator, not a capacity benchmark. A self-hosted classifier runs in its own service. The job uses Polyaxon's normal tracking and artifact collection utilities; ensure those are enabled in your deployment.

Launch the baseline and open its record:

polyaxon run -f evaluate.yaml
polyaxon ops dashboard

Compare an experiment wave before asking for another idea

Register the reviewed evaluator as YOUR_ORG/pronunciation-eval:v1 in the Component Hub. A mapping gives each explicit candidate its own child run. This wave compares the baseline with three harness changes on 行 while evaluating the complete development manifest:

version: 1.1
kind: operation
name: pronunciation-wave-01
hubRef: YOUR_ORG/pronunciation-eval:v1
cache:
  disable: true
params:
  repeats:
    value: 3
  max_calls:
    value: 2000
matrix:
  kind: mapping
  concurrency: 2
  values:
    - proposal: '{"mode":"definitions","target":"all","limit":4,"criteria":{}}'
    - proposal: '{"mode":"lexicon","target":"行","limit":4,"criteria":{}}'
    - proposal: '{"mode":"boundary","target":"行","limit":4,"criteria":{}}'
    - proposal: '{"mode":"chain","target":"行","limit":4,"criteria":{}}'

For other characters the target-specific candidates use the same definitions baseline. Add the phrase-table and label-only baselines as separate rows when establishing the study. A larger reasoning model is another baseline if needed; give its adapter exactly the same case identities and scorer, including explicit alignment checks for sequence output.

After independent slice studies, materialize a candidate routing map. On a corpus containing both 行 and 重, this configuration uses two calls selectively for 行 and boundary-filtered evidence for 重:

{
  "mode": "definitions",
  "target": "all",
  "routes": {
    "行": {"mode": "chain", "limit": 4},
    "重": {"mode": "boundary", "limit": 2}
  }
}

Submit the complete map as another candidate and reevaluate the full development set. Separate improvements on two slices are evidence for those changes individually, not yet evidence for their combined deployment.

Matrix concurrency bounds child operations. It does not impose a provider-wide request or currency budget. The example job submits requests sequentially, and a chained decision can make two requests. Before admitting a wave, your controller reserves a worst-case allowance for all children and repetitions. Include model calls used to propose candidates in that allowance. Job timeouts and call counts are useful limits, but neither alone is a precise currency cap.

For parameter-only variants, grid or native TPE can search a declared space. A proposal about new source code or a new evidence mechanism belongs in the research application, not inside the numerical optimizer.

Give the research policy a small, useful tool surface

The policy can be a person, a script, or a coding agent. Give it development failures and a bounded set of tools:

A cheap probe answers a structural question before a paid scoring pass. Does the dictionary contain a covering word? Does the segmenter preserve its boundaries? Are two readings being merged by normalization? A few inspected cases can reject a broken mechanism; they cannot establish a quality improvement.

Use SandboxClient to execute the reviewed inspection program. Keep the client bound to the selected workspace; the model does not choose its project or run. The coding-agent tutorial shows file/process integration. A sandbox exposes execution inside its service container; use deployment controls appropriate to the code and mounts it can access.

For a concrete probe, start a sandbox-enabled service using the evaluator image, a long-running command such as sleep 3600, and only the development connection. Disable the in-container auth context with plugins.auth: false; management credentials stay in the host. The sandbox quick start supplies the service setup and readiness procedure. This host-side call inspects the first case with the same evidence builder and makes no classifier request:

import json
import os

from polyaxon.client import SandboxClient


probe = """
import json
from pathlib import Path
from evaluate import prepare_lexicon, word_evidence
cases = json.loads(Path('/data/dev/cases.json').read_text(encoding='utf-8'))
lexicon = json.loads(Path('/data/dev/lexicon.json').read_text(encoding='utf-8'))
print(json.dumps(word_evidence(cases[0], prepare_lexicon(lexicon), False, 4), ensure_ascii=False))
"""
with SandboxClient(owner=os.environ["RESEARCH_OWNER"],
                   project=os.environ["RESEARCH_PROJECT"],
                   run_uuid=os.environ["PROBE_RUN_UUID"]) as sandbox:
    result = sandbox.process.exec(command=["python", "-c", probe],
                                  workdir="/app", timeout_ms=30000)
    print(json.dumps({"exit_code": result.exit_code, "timed_out": result.timed_out,
                      "stdout": result.stdout, "stderr": result.stderr}))

The probe is fixed application code. Validate case selection and bound returned output before exposing it as an agent tool; do not replace it with unrestricted model-generated commands.

For configuration proposals, the host submits a frozen component instead of asking the model to emit a shell command:

import hashlib
import json

from polyaxon.client import RunClient


def submit_approved_proposal(owner, project, proposal, protocol_digest):
    # Called only after the host validates the proposal and reserves its budget.
    encoded = json.dumps(proposal, sort_keys=True, separators=(",", ":"))
    identity = json.dumps({"proposal": proposal, "protocol": protocol_digest},
                          sort_keys=True, separators=(",", ":"))
    candidate_id = hashlib.sha256(identity.encode()).hexdigest()
    client = RunClient(owner=owner, project=project)
    run = client.create_from_hub(
        component=owner + "/pronunciation-eval:v1",
        name="pronunciation-" + candidate_id[:12],
        params={"proposal": encoded, "repeats": 3, "max_calls": 2000},
    )
    return {"candidate_id": candidate_id, "protocol": protocol_digest,
            "run_uuid": run.uuid}

The SDK method submits an operation; this helper is not a durable campaign controller. The host must persist the proposal-to-run mapping and reconcile uncertain submissions before retrying. protocol_digest identifies the frozen evaluator, model, data, environment and normalization record, so the same configuration under a different protocol is a different candidate.

Put a finite envelope around the host loop: for example, three waves, at most four candidates per wave, a separate limit on proposer calls, and a total deadline. Reserve each child's request ceiling before submission, release unused reservations after accounting, and stop when the next wave cannot be admitted. These are application limits; a prompt asking the agent to stay within budget is not enforcement.

After a completed wave, group the remaining development errors by character, reading, word evidence, token-boundary agreement, and invalid response. Propose one mechanism to investigate next. Retain rejected and inconclusive proposals with their reasons so the policy can use earlier evidence without repeatedly paying for the same experiment.

Start with configuration changes. An agent that invents a new parser or source implementation needs a new immutable candidate artifact, reviewed execution permissions, and evaluation outside the code it can modify. It must not edit the expected labels or scoring program to improve its result.

Measure regressions and separate improvement from variation

Compare candidate and baseline on the same cases and repetitions. Let B count baseline mistakes the candidate fixes, and C count baseline-correct cases the candidate breaks. The net error reduction is B − C. Keep both counts and the affected case IDs; the difference alone hides who pays for the change.

Use the full development manifest before retaining a candidate, not only the few cases that motivated it. Check per-character and rare-reading slices, then transfer a promising mechanism to development characters it did not target. Combining two successful changes creates another candidate that needs its own comparison.

Useful ablations include:

These are experiments, not a list of changes assumed to help. More context can add cost without correcting a decision. A new feature can fix its target slice while worsening another. Record both outcomes.

Repeat the baseline and candidates to observe response variation under the declared service load. The minimal job reports client-observed latency including first calls; use a separate, recorded warm-up workload and randomized or interleaved evaluation order before making steady-state latency claims. On a shared self-hosted service, parallel jobs introduce queueing, so a concurrent quality wave is not an isolated serving-latency benchmark.

Repeated calls on one sentence do not create new independent language examples. Report repetitions separately and choose a paired uncertainty procedure consistent with the document grouping and metric. Dror et al. discuss why the experimental setup determines the appropriate significance test. Predeclare the minimum useful improvement and regression limits; do not import an error-count threshold from another corpus.

Retain a candidate only when evaluation is complete and comparable, the improvement meets the declared evidence threshold, no protected slice crosses its regression limit, and the latency/cost guardrails hold. Record smaller or unstable changes as inconclusive. The next proposal should be informed by that outcome without turning an inconclusive score into an accepted result.

Confirm one frozen candidate outside the research loop

Run promotion under a reviewer-controlled component or project that can read the promotion connection. The research workspace has neither those cases nor credentials that can retrieve them. The evaluator's auth context follows the submitting user's access, so mounting only development files is insufficient if the agent also holds broader API credentials.

Reuse the evaluator image digest and exact proposal JSON. In the reviewer-owned component, replace the development connection and the --cases path with the protected manifest's connection and path. Keep the lexicon and scoring contract fixed. The agent's evaluation tool cannot select this component or change its dataset arguments.

Limit and count promotion accesses. If their results influence another proposal, they have become part of development selection. Keep final-assessment cases and results outside proposal context. For final assessment, run the fixed candidate without further selection on those results, then retain the full report with the release decision.

The release record should identify the selected request/evidence configuration, candidate artifact, model and server revision, dataset and evaluator digests, repetitions, guardrails, and the qualifying run. Approval updates the application's versioned configuration; a good score does not automatically deploy a model or change production routing.

Report the system and the cost of finding it

Use run comparison to filter to compatible protocol identities and display the following evidence. The evaluation job writes the raw request and case records to artifacts, not just a headline score.

For hosted inference, calculate currency cost from actual billed usage and a recorded pricing basis. Missing usage is unknown, not zero. For self-hosted Qwen, input-token counts are not a bill: account for service allocation and utilization, including idle time and shared capacity. Keep the cost of running the selected harness separate from the cost of discovering it.

Plot complete candidate results alongside the retained frontier. Keep failed and rejected runs visible, and expose cost and per-character reports alongside the score. The familiar shape is useful, but it does not supply measurement evidence by itself:

Constructed progress example reused from the autoresearch design note; these are not pronunciation or model benchmark results. Generate the corresponding view from your own completed run records.

Once a candidate is qualified, register its immutable component/configuration revision and schedule regression evaluations against reviewed case versions. Use the new failures to create the next development set while preserving the previous assessment record. Stop temporary classifier and research-workspace services when the study ends; keep the run records and decision artifacts.

The first useful result is a reproducible baseline and a complete candidate comparison. A frozen model can then remain a stable experimental instrument while the research loop changes the evidence, questions, and routing around it.