Kubernetes taints and tolerations for ML workloads

Taints repel Pods from nodes. Tolerations allow a Pod to remain eligible despite a matching taint. Together they protect specialized or constrained capacity from workloads that do not belong there.

The most important limitation is in the wording: a toleration permits placement; it does not attract the Pod to that node. GPU and other ML placement policies usually need a toleration plus a positive rule such as node affinity and an explicit resource request.

Understand the three effects

A taint has a key, optional value, and effect. Kubernetes supports:

The Kubernetes taints and tolerations documentation defines how multiple taints are filtered against a Pod's tolerations. Any remaining unignored taint can affect placement or execution.

Inspect current taints before making a change:

kubectl --context production get nodes \
  -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints'
kubectl --context production describe node gpu-worker-1

Protect specialized capacity

A platform team might reserve GPU nodes with a taint such as:

kubectl --context production taint node gpu-worker-1 \
  dedicated=accelerator:NoSchedule

The matching workload needs a toleration:

tolerations:
  - key: dedicated
    operator: Equal
    value: accelerator
    effect: NoSchedule

This allows the Pod onto the node, but it can still schedule elsewhere. Pair it with an explicit accelerator request and node affinity based on labels your platform controls:

resources:
  limits:
    nvidia.com/gpu: 1
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: platform.polyaxon.com/accelerator-pool
              operator: In
              values: [gpu]

Use real labels and extended-resource names from your cluster. Do not infer hardware capability from an arbitrary taint value.

Treat NoExecute as an eviction policy

NoExecute affects running Pods, not only future scheduling. A toleration can include tolerationSeconds, allowing a Pod to remain for a bounded time after the taint appears:

tolerations:
  - key: capacity.polyaxon.com/interruption
    operator: Exists
    effect: NoExecute
    tolerationSeconds: 120

This is not a checkpoint guarantee. The application needs to handle termination signals, persist progress during normal execution, and tolerate abrupt loss. Choose the time from the actual notice and shutdown behavior of the infrastructure; do not copy the example value into production without measuring it.

Some node-condition taints are managed automatically by Kubernetes. Avoid removing them to force a workload onto an unhealthy node. Diagnose the underlying memory, disk, network, or readiness condition instead.

Separate repulsion, attraction, and allocation

Use each mechanism for its intended decision:

• Taints and tolerations: which Pods may use a node despite a restriction?
• Node selectors or affinity: which nodes should or must a Pod use?
• Resource requests: does the node have the required CPU, memory, GPU, or device?
• Queues and quotas: when may the workload consume shared capacity?
• Priority and preemption: which admitted work is more important under contention?

A toleration does not reserve capacity, enforce fair sharing, or prove that a GPU is available. Combining unrelated policy in one label makes scheduling harder to explain.

Encode policy through Polyaxon presets

Polyaxon scheduling presets can apply common tolerations, affinity, resources, and security settings without asking every user to reproduce cluster-specific YAML. Keep presets narrow and named by intent, such as “GPU batch” or “interruptible training,” rather than by a temporary node-pool implementation.

Use node scheduling for the supported configuration paths. Review changes centrally because one broad toleration can make many workloads eligible for protected capacity.

Diagnose Pending Pods systematically

When a Pod remains Pending, inspect its events and effective specification:

kubectl --context production --namespace ml-team describe pod training-run-abc123
kubectl --context production --namespace ml-team get pod training-run-abc123 -o yaml

Compare every node constraint: taints, affinity, resources, topology, volume binding, and queue admission. Fix the actual unsatisfied condition rather than adding an Exists toleration for every taint.

Monitor how often workloads are unschedulable by reason and how much protected capacity is used by its intended class. A placement policy is successful when it is both enforced and explainable: operators can state why the Pod was eligible, why it selected that pool, and what happens when the capacity is unavailable.