Polyaxon v3 is coming →
Axolotl

Fine-tune with Axolotl on Polyaxon

Axolotl keeps model, dataset, LoRA, trainer, and distributed settings in its own YAML. Polyaxon keeps that file and its validated image attached to the infrastructure, permissions, run history, and checkpoint lifecycle.

Outcome

A reusable Axolotl component whose framework configuration can change independently from platform policy.

Start the guideScaling documentation

Execution responsibilities

Polyaxon resolves

The component pins the image, inputs, secrets, resources, queue, and output contract.

Kubernetes places

The cluster schedules the requested accelerator topology and enforces namespace policy.

The framework trains

The training stack owns optimization while Polyaxon records status, logs, metadata, and artifacts.

Prerequisites

  • A reviewed Axolotl configuration stored with the code revision.
  • A pinned Axolotl image compatible with the cluster driver and selected attention kernels.
  • Connections for gated models, datasets, tracking, and checkpoint storage.

Run and validate

  1. 1

    Wrap the Axolotl configuration

    Mount or package the Axolotl YAML with the code and let the component define only the infrastructure contract.

    version: 1.1kind: componentname: axolotl-trainrun:  kind: job  connections: [hf-token, training-data, checkpoints]  container:    image: axolotlai/axolotl:0.16.1    command: ["axolotl", "train", "/workspace/axolotl.yml"]    resources:      requests:        cpu: "16"        memory: 96Gi      limits:        nvidia.com/gpu: "4"

    Four GPUs are an example shape, not a universal requirement. Size the run from the chosen model, precision, context length, and Axolotl strategy.

  2. 2

    Submit the operation

    Run the component through the target Polyaxon project, queue, preset, and approval path.

    polyaxon run -f axolotl.yaml
  3. 3

    Inspect training and checkpoints

    Verify the resolved Axolotl config, GPU count, dataset revision, loss curve, checkpoint path, and final adapter or model artifact.

    polyaxon ops dashboard

Production checklist

Pin the container image, framework versions, model revision, dataset revision, and training configuration.
Resolve registry, dataset, model, and tracking credentials through scoped Polyaxon connections.
Persist checkpoints and final artifacts outside the pod, and define resume behavior before a long run.
Set requests, limits, node selectors, tolerations, queue policy, timeout, and retry ownership explicitly.

Troubleshooting

The pod remains Pending

Inspect quota, queue capacity, node selectors, taints, topology, and the requested accelerator count.

The process runs out of memory

Check model precision, sequence length, batch size, checkpointing, optimizer state, sharding, and actual device memory.

The run cannot read or save

Verify the referenced connection, secret scope, mounted path, object-store permissions, and available storage.

Sources

Official platform, library, model, and Polyaxon references used by this guide.

Continue

Compare AI platformsExplore MLOps