Turn LLM experiments into a reusable engineering knowledge base

Someone asks why the documentation assistant uses a hybrid retriever. The answer should be more useful than a link to a vendor article or “it worked better when we tried it.” A teammate should be able to find the compared runs, inspect the query set, see which cases improved, and understand the decision's limits.

Polyaxon's run queries, comparison dashboard, and artifact lineage provide the execution records for that knowledge base. The missing ingredient is usually a consistent convention for recording the question and decision alongside the measurements.

Organize evidence around a decision

Consider this proposed experiment: adding a reranker may let a documentation assistant use less context without reducing answer correctness. Record the hypothesis before running candidates, together with the query-set revision, source snapshot, baseline, and latency constraint.

An external engineering account can explain why the idea is worth investigating. Cite its original author and source in the decision note, distinguish reported results from your own hypothesis, and record important differences from your workload. Do not turn a third party's performance claim into your application's expected improvement.

Keep external research in your team's existing document catalog or wiki. Use Polyaxon to preserve the local experiments that determine whether an idea works for your application. Run search is structured metadata search; it is not automatically a semantic search engine over attached papers or reports.

Define a small metadata convention

Use component inputs for configurable experimental choices and fixed tracked outputs for revision identifiers and decisions. Put detailed reasoning and case-level evidence in artifacts.

Use metadata logging consistently across components. Calling the same measurement accuracy, score, and quality in three runners makes discovery harder and can conceal incompatible definitions. Record the definition and denominator with the report.

Find comparable runs from the CLI

Suppose your evaluation jobs log an acceptance_rate metric and declare a retrieval_strategy input. The following query finds up to ten jobs above an illustrative acceptance threshold and displays their settings and score:

polyaxon ops ls -p retrieval-evaluations \
  -q "kind: job, metrics.acceptance_rate: >0.9" \
  -s "-metrics.acceptance_rate" \
  -l 10 -io \
  -c "uuid,in.retrieval_strategy,out.acceptance_rate"

Replace the project and fields with your own convention. The threshold is an example filter, not a production quality standard. A high score only makes a run worth inspecting; confirm the corpus, cases, evaluator, and operating conditions before comparing it with another run.

The CLI query documentation covers filters, sorting, selected columns, pagination, and CSV export. The client interface exposes the same discovery workflow for a report generator or internal catalog. Iterate through pages when you need the complete population rather than a shortlist.

Retrieve the evidence behind a score

Once you have a run ID, inspect its configuration and download its evaluation report. These commands assume the runner saved evaluation-report.json in its outputs directory, as in the tracking example:

polyaxon ops get -p retrieval-evaluations -uid RUN_UUID
polyaxon ops artifacts -p retrieval-evaluations -uid RUN_UUID -f outputs/evaluation-report.json

Replace RUN_UUID with the selected run. In the UI, the lineage view exposes input and output artifacts and related operations. Use it to identify preparation and scoring dependencies instead of inferring them from similar run names.

Artifact references and readable files are different things. A reference can remain after a remote object changes or becomes inaccessible. Keep immutable revisions where possible, retain content digests, and choose storage and retention policies that preserve the evidence your team needs.

Save the decision as part of the experiment

The following script attaches an existing decision note to the current Polyaxon run. Use it from a reporting job with the polyaxon package installed and a reviewed decision.md file available in the working directory.

from pathlib import Path

from polyaxon import tracking

tracking.init()
decision = Path("decision.md").read_text(encoding="utf-8")
output_path = Path(tracking.get_outputs_path("decision.md"))
output_path.write_text(decision, encoding="utf-8")
tracking.log_file_ref(path=str(output_path), name="engineering-decision")

Include the baseline and candidate run IDs in the note, the evaluation population, the selected configuration, known regressions, unresolved uncertainties, and an owner. State the actual conclusion narrowly: “selected hybrid retrieval for this documentation snapshot and query set,” rather than “hybrid retrieval is always better.”

Keep a rejected experiment discoverable too. A negative result with a clear workload and constraint can prevent another engineer from repeating the same work. A failed run may also contain useful evidence, but distinguish an unsuccessful execution from a completed experiment that rejected a hypothesis.

Revisit conclusions when their inputs change

Choose a review trigger for consequential decisions: a new embedding model, changed access rules, a different workload mix, or a major corpus revision. Link the new comparison to the earlier decision rather than replacing its report.

Use a reusable evaluation component and a bounded retrieval experiment matrix to repeat the comparison. Keep the question and measurement procedure stable enough to explain what changed.

The result is a practical engineering memory: searchable configurations, inspectable artifacts, and decisions connected to execution evidence. External examples continue to supply ideas, while your own runs establish what the team has actually demonstrated.