Polyaxon v3 is coming →
Google GKE

Validate a GKE GPU cluster with Polyaxon

GKE can manage the GPU node lifecycle and driver installation, while Standard clusters also support more platform-owned configurations. Polyaxon keeps the selected machine family, driver mode, topology, and identity policy in queues and presets rather than application code.

Outcome

A GKE capacity class exposed through Polyaxon with an auditable readiness operation for its GPU, storage, identity, and network path.

Start the guideConnect a compute cluster to Polyaxon

Infrastructure responsibilities

GKE manages the Kubernetes service

GKE operates the control plane and can manage GPU node provisioning and driver installation, depending on the chosen mode.

Your platform team defines capacity

Node pools, machine families, driver policy, networking, storage, identity, labels, taints, and upgrade choices remain explicit platform decisions.

Polyaxon connects the compute cluster

Installing Polyaxon Agent (compute cluster) connects this GKE cluster to the control plane. Its queues and presets apply placement, connections, permissions, and lifecycle controls.

Prerequisites

  • Regional quota and supported GKE Standard or Autopilot capacity for the selected GPU machine family.
  • A node pool or workload class with reviewed driver mode, topology, labels, taints, and autoscaling behavior.
  • The network, GPUDirect mode, firewall rules, storage, and service account required by the workload.
  • The cluster connected to Polyaxon by installing Polyaxon Agent (compute cluster), with namespace policy, storage connections, and a queue targeting the GPU pool.

Validate the infrastructure

  1. 1

    Target the GKE accelerator pool

    Keep machine-family and fabric labels inside a platform-owned preset and run a diagnostics image before scheduling training.

    version: 1.1kind: componentname: gpu-readinessrun:  kind: job  container:    image: nvidia/cuda:12.8.1-base-ubuntu22.04    command: ["bash", "-lc"]    args: ["nvidia-smi"]    resources:      limits:        nvidia.com/gpu: "1"

    Keep GKE placement in the selected queue or preset. This published CUDA image checks GPU visibility only; pin a tag or digest compatible with your driver. Test NCCL or GPUDirect separately when that fabric is configured.

  2. 2

    Submit the operation

    Run the component through the target Polyaxon project, queue, preset, and approval path.

    polyaxon run -f gcp-readiness.yaml -q agent-name/queue-name
  3. 3

    Inspect the resolved GKE placement

    Confirm the selected node pool, GPU visibility, storage access, service DNS, and workload identity from the running operation.

    polyaxon ops dashboard

Production checklist

Record whether GKE or your platform team owns driver installation and node lifecycle for each capacity class.
Separate general-purpose, training, and inference pools with explicit labels, taints, quotas, and Polyaxon queues.
Validate the driver, runtime, storage, DNS, identity, and network path after GKE or node-image upgrades.
Treat each GPU machine family, driver mode, and network topology as a separately validated capacity class.

Troubleshooting

The GPU resource is missing

Check whether the node pool is running, the selected GKE driver mode completed successfully, and nvidia.com/gpu appears in node capacity.

The operation lands on the wrong pool

Compare the queue or preset selector with the node pool labels, taints, tolerations, and requested GPU resource.

The operation cannot access data or Google APIs

Verify the Kubernetes service account, Workload Identity binding, storage connection, network policy, and required egress.

Sources

Official platform, library, model, and Polyaxon references used by this guide.

Continue

Compare AI platformsExplore GPU orchestration