Polyaxon v3 is coming →
Tenstorrent

Run Polyaxon workloads on Tenstorrent

Tenstorrent support starts below Polyaxon: the cluster must expose healthy devices through the operator or device-integration path chosen by the platform team. Polyaxon then schedules a compatible component using a reviewed preset and records its execution.

Outcome

A Tenstorrent readiness component template that delegates the evolving device-acquisition details to a centrally managed preset.

Start the guideScaling documentation

Infrastructure responsibilities

Kubernetes discovers

Drivers and a vendor device integration expose healthy accelerator capacity to the scheduler.

Polyaxon places

A component or preset requests the custom resource and targets the compatible node pool and image.

The workload verifies

A smoke test proves device access before training or serving is promoted to that hardware pool.

Prerequisites

  • Tenstorrent hardware with supported kernel driver, firmware, HugePages or IOMMU configuration, and user-mode software.
  • A Kubernetes device integration or DRA operator selected and maintained by the platform team.
  • A validated Tenstorrent image plus a Polyaxon preset that injects the exact resource claim, mounts, labels, and security context.

Validate the infrastructure

  1. 1

    Use the platform-owned Tenstorrent preset

    Keep fast-moving resource-claim and device-mount details outside user components; the component declares the workload and diagnostic command.

    version: 1.1kind: componentname: tenstorrent-readinessrun:  kind: job  environment:    nodeSelector:      workload.polyaxon.com/accelerator: tenstorrent  container:    image: ghcr.io/tenstorrent/tt-metal/tt-metalium-ubuntu-22.04-release-amd64:latest-rc    command: ["tt-smi", "-ls"]

    Apply a platform-owned preset that injects the device claim or mounts required by your installed Tenstorrent integration. There is no universal Kubernetes resource key to put in this portable component; pin the image digest before production.

  2. 2

    Submit the operation

    Run the component through the target Polyaxon project, queue, preset, and approval path.

    polyaxon run -f tenstorrent-readiness.yaml
  3. 3

    Verify the device inventory

    Confirm the allocated board is visible to `tt-smi`, the runtime can execute a small framework example, and the pod uses the intended security and HugePages policy.

    polyaxon ops dashboard

Production checklist

Pin a compatible driver, device integration, container runtime, framework image, and firmware combination.
Publish accelerator selection through centrally managed Polyaxon presets instead of user-specific labels.
Track allocatable devices, hardware health, utilization, memory, thermals, and workload failures.
Validate one device, one node, and multi-node communication separately before production adoption.
Version the preset with the operator, DRA API, driver, firmware, HugePages, and container-image compatibility set.

Troubleshooting

The resource does not appear

Inspect the node driver, firmware, device integration, kubelet registration, node labels, and allocatable resources.

The container cannot access the device

Verify the requested custom resource, runtime class, permissions, device mounts, and image compatibility.

Performance is unexpectedly low

Check topology, NUMA placement, clocks, memory pressure, host-device transfer, communication libraries, and framework build.

Sources

Official platform, library, model, and Polyaxon references used by this guide.

Continue

Compare AI platformsExplore GPU orchestration