Check Kubernetes version compatibility

“What Kubernetes version are we running?” sounds like a one-command question. In practice, a cluster has several relevant versions: the local kubectl client, the API server, each node's kubelet, and the APIs used by installed controllers and workload manifests.

Checking all of them matters before an upgrade, when a command behaves differently across environments, or when a controller works in one cluster but rejects the same resource in another.

Check the client before the cluster

The local client can be inspected without contacting a cluster:

kubectl version --client -o yaml

This confirms which binary your shell is invoking. It is especially useful on machines with package-manager installations, downloaded binaries, IDE integrations, or multiple paths.

Record the complete semantic version rather than only the minor release. Patch releases can include fixes that affect authentication, output formatting, or client behavior.

Check the selected API server

Query the cluster with an explicit context:

kubectl --context production version -o yaml

The result includes client and server information. The explicit context is important: an accurate answer for the wrong cluster is still operationally dangerous. If you regularly work across clusters, follow the safeguards in Manage Kubernetes contexts safely.

Client and server versions do not need to be identical, but they must stay within the supported Kubernetes version-skew policy. Check that policy for the Kubernetes release you operate instead of relying on a remembered rule; the allowed relationship can vary by component and upgrade state.

Inventory node versions

Control-plane upgrades and node upgrades are separate operations. Inspect the kubelet version reported by every node:

kubectl --context production get nodes \
  -o custom-columns='NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion,RUNTIME:.status.nodeInfo.containerRuntimeVersion,OS:.status.nodeInfo.osImage'

A temporary version mix can be expected during a rolling upgrade. A long-lived outlier is different: it may indicate a failed node-pool rollout, an autoscaling group on an old image, or a node that rejoined with stale configuration.

For ML infrastructure, group the inventory by node pool and accelerator class. GPU nodes often follow a separate maintenance path because their operating system image, container runtime, driver, device plugin, and Kubernetes version must work together.

Check APIs, not only version numbers

Two clusters with the same Kubernetes version can expose different APIs because feature gates, extensions, admission policies, and custom resources differ. Confirm the resources your platform needs:

kubectl --context production api-resources
kubectl --context production get customresourcedefinitions

Before changing a manifest, ask three questions:

1. Is its apiVersion served by the destination cluster?
2. Is that version still accepted for new writes?
3. Do installed controllers support the object's fields and semantics?

This is why a compatibility inventory should include operators, admission controllers, storage and network plugins, GPU components, and workload CRDs—not only the API server.

Use discovery to explain errors

Errors such as “the server doesn't have a resource type” and “no matches for kind” often point to discovery or installation differences. First verify the active context, then inspect the API group directly:

kubectl --context production api-resources --api-group=batch
kubectl --context production explain job.spec

If a custom workload exists in staging but not production, compare the installed CRD and controller version before rewriting the workload. If kubectl explain does not show a field, the destination API schema may differ from the one used to author the manifest.

Build an upgrade compatibility record

Before an upgrade, capture a small, reviewable record:

Validate the target version in a representative non-production cluster. Run real Polyaxon workloads that exercise artifact connections, scheduling, termination, services, and distributed execution. A cluster that reports healthy nodes can still break a platform integration at the API or admission layer.

Keep workload behavior portable

Polyaxon separates the workload specification from environment-specific scheduling and connections. Scheduling presets can carry cluster-specific settings without duplicating the whole workload, while versioned operation specifications keep the execution definition with the project.

That separation reduces upgrade risk, but it does not eliminate compatibility work. Treat version information as part of the platform's operating evidence: explicit, captured before change, and verified with representative workloads after change.