Request GPUs by capability with Kubernetes DRA

White Kubernetes wheel above a blue abstract platform

A training recipe needs a GPU with enough memory for its model, but the cluster contains several accelerator generations. Requesting one GPU answers how many devices the workload needs. It does not, by itself, describe which devices are suitable.

Dynamic Resource Allocation (DRA) gives Kubernetes a richer device request model. A workload can reference a claim that selects devices using capabilities published by a driver. The core mechanism is stable in Kubernetes 1.35, although individual DRA extensions have their own feature states. A compatible driver and platform configuration are still prerequisites. Kubernetes DRA allocation guide.

The useful shift for an ML platform is from remembering a particular node pool to expressing a hardware requirement. This article develops that distinction; preparing for new GPU generations covers the broader migration work.

Start with the workload requirement

Imagine a platform with 24 GiB and 80 GiB accelerators. A training recipe needs at least 48 GiB per device. A smaller evaluation job works on either type. These are illustrative requirements, not measured memory budgets.

If both workloads select the same expensive node pool, the evaluation job unnecessarily competes with training. If training requests an undifferentiated GPU, it may receive an unsuitable device unless another placement constraint prevents it.

Write down the actual contract before choosing a selector:

Requirement	Evidence to preserve
Enough accelerator memory	Peak use for the model, batch size, precision, and optimizer configuration
Compatible execution stack	Driver, runtime, framework, and kernel requirements
Multiple communicating devices	Device count plus the required interconnect and placement constraints
Acceptable alternative hardware	Correctness and performance results for each supported class

A memory threshold cannot establish software compatibility or communication performance. Treat it as one requirement among several.

Understand the objects behind an allocation

The driver publishes device information through ResourceSlice objects. A cluster administrator defines a DeviceClass; a namespaced ResourceClaim asks for devices from a class. Scheduling and driver preparation connect the selected devices to the consuming workload. How DRA works.

That division creates three useful ownership boundaries: the driver describes what exists, the platform defines an approved class, and the application states what it needs. An application team should not have to reverse-engineer the hardware inventory for every run.

A ResourceClaimTemplate creates a claim for each consuming Pod. A directly referenced claim has a separately managed lifetime and can support sharing arrangements. Neither arrangement automatically partitions GPU memory or provides fractional compute isolation; those behaviors depend on the device and driver.

Express a capability request

The following template illustrates one device with at least 48 GiB of advertised memory. It assumes an existing training-gpu DeviceClass and a driver that publishes a memory capacity under driver.example.com. That domain and attribute schema are placeholders: substitute the actual driver's documented keys and supported types.

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: training-accelerator
  namespace: ml-team
spec:
  spec:
    devices:
      requests:
        - name: accelerator
          exactly:
            deviceClassName: training-gpu
            allocationMode: ExactCount
            count: 1
            selectors:
              - cel:
                  expression: >-
                    device.capacity["driver.example.com"].memory.compareTo(quantity("48Gi")) >= 0

Device selectors use CEL expressions over the driver's published data. The available attributes are not universal GPU field names. Check the ResourceClaim API and the installed driver's schema before adapting the expression.

This Pod-spec fragment connects the generated claim to the training container. Merge it into a complete workload with your approved image, application command, host CPU/memory requests, and other required settings; it is not a standalone training manifest.

spec:
  resourceClaims:
    - name: training-device
      resourceClaimTemplateName: training-accelerator
  containers:
    - name: trainer
      # Keep the rest of this container's existing configuration.
      resources:
        claims:
          - name: training-device

Do not blindly add a legacy nvidia.com/gpu request alongside the claim. Decide which allocation mechanism the driver and workload integration use so that the request does not accidentally demand two separate allocations.

Follow the request through each stage

When a workload waits, inspect the class, claim, Pod, and events together:

kubectl get deviceclasses
kubectl get resourceclaims -n ml-team
kubectl get pods -n ml-team

Then describe the relevant claim and Pod by their returned names. An unsatisfied selector calls for different evidence from a claim that has an allocation but whose device preparation fails on the node. Record the claim reference with the run; a generic “GPU unavailable” message loses this distinction.

For a platform trial, compare two recipes: the demanding training job and the smaller evaluation job. Confirm that the former excludes unsuitable devices and the latter retains its intended choices. Separately measure application correctness, startup time, and runtime. Successful allocation proves placement, not model performance.

Apply the contract to Polyaxon workloads

Polyaxon makes hardware selection reusable across training, evaluation, and interactive development. A platform team can package resource requests, placement settings, and connections in scheduling presets, so the same training recipe can use different hardware profiles without embedding cluster-specific settings in application code.

For example, a team could offer separate profiles for a GPU debugging sandbox and a larger training run. Polyaxon EE and Cloud can combine presets and queues into a catalog of compute environments, with routing and concurrency controls. DRA can make the underlying device selection more expressive when the deployed workload integration supports creating claims and attaching them to Pods.

Start with one validated device class and one representative Polyaxon recipe. Record the allocated device and application environment with the run to compare eligible hardware against actual results. Confirm claim support in the deployed Polyaxon schema, controller, and driver; the native DRA fragment above is not a drop-in preset. For GPU allocation through extended resources, use Polyaxon's existing resource scheduling.