Hold Pods until they are ready for scheduling

A training Pod exists, but the dataset release it needs has not passed validation. Sending it into placement now creates work for the scheduler even though an external prerequisite remains unresolved.

Kubernetes scheduling gates let a Pod exist without being eligible for scheduling yet. The mechanism has been stable since Kubernetes 1.30. Gates are declared when the Pod is created and removed as prerequisites are satisfied; new gates cannot be added afterward. Pod scheduling readiness documentation.

The important design decision is who removes a gate and what evidence authorizes that action. Kubernetes does not interpret a gate named dataset-ready as a request to inspect a dataset.

Put the wait at the right stage

A workload can wait before a Pod exists, before placement, during container preparation, or before accepting traffic. These stages have different costs and owners.

An init container runs after placement, before the application container. A readiness probe determines whether a running container is ready to serve traffic. Neither substitutes for an external prerequisite that should keep the Pod out of scheduling.

Avoid a circular dependency: a gated Pod cannot run its own init container to satisfy the condition that would remove its scheduling gate.

Observe a single gate

The following disposable example requires Kubernetes 1.30 or later, permissions to create and patch Pods, and access to the Python image. It represents an external dataset approval with a manually removed gate. It does not validate any data.

Save it as scheduling-gate-demo.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: scheduling-gate-demo
---
apiVersion: v1
kind: Pod
metadata:
  name: waiting-worker
  namespace: scheduling-gate-demo
spec:
  restartPolicy: Never
  schedulingGates:
    - name: platform.example.com/dataset-ready
  containers:
    - name: worker
      image: python:3.12-slim
      command:
        - python
        - -c
        - "print('The scheduling gate was removed; the worker can now start.')"
      resources:
        requests:
          cpu: "100m"
          memory: "64Mi"
        limits:
          cpu: "500m"
          memory: "128Mi"

The image tag keeps this learning fixture simple; use your approved digest for a reproducible workload. Create it and inspect the stored Pod:

kubectl apply -f scheduling-gate-demo.yaml
kubectl get pod waiting-worker -n scheduling-gate-demo -o yaml
kubectl get pod waiting-worker -n scheduling-gate-demo

Before gate removal, expect the gate to remain in the spec, no assigned node, and a SchedulingGated status in the tabular display. The underlying Pod phase is Pending. A gated Pod is intentionally withheld; do not treat this state as evidence of insufficient GPU capacity.

Remove only the gate you own

In this single-gate fixture, the operator stands in for a controller that has verified the external prerequisite. The JSON Patch first checks that the entry is the intended gate, then removes it:

kubectl patch pod waiting-worker -n scheduling-gate-demo --type=json \
  --patch '[{"op":"test","path":"/spec/schedulingGates/0/name","value":"platform.example.com/dataset-ready"},{"op":"remove","path":"/spec/schedulingGates/0"}]'

The patch's test operation protects this example from removing a different entry if the array has changed. With several gates, a production controller must locate its own entry in fresh state and handle conflicts. It should not clear the entire list on behalf of other controllers. See the kubectl patch reference for patch formats.

Inspect the Pod again and read its logs once it runs:

kubectl get pod waiting-worker -n scheduling-gate-demo -o wide
kubectl logs waiting-worker -n scheduling-gate-demo

Removing the last gate makes scheduling possible; it does not guarantee immediate placement or successful image startup. If the Pod continues waiting, investigate its new conditions and events. These steps describe expected behavior, not a recorded execution of this fixture.

Clean up the dedicated namespace afterward:

kubectl delete namespace scheduling-gate-demo

Give every gate an operational owner

For a real dataset-release controller, retain the immutable dataset identifier, the approval evidence, the gate's creation time, and the identity that removed it. Define what happens when validation fails or takes too long. A gate that remains forever is a hidden backlog unless someone monitors its age and reason.

Also decide how the workflow records cancellation. Removing a gate to “unstick” an invalid workload defeats its purpose. The controller should propagate a meaningful failure or cancellation to the owning operation and clean up according to that operation's lifecycle.

This mechanism is not a capacity reservation, quota allocator, or a way to pause a running Pod. It expresses a prerequisite before scheduling. Keep those boundaries visible in dashboards and incident reports.

Use the existing workflow boundary in Polyaxon

Polyaxon lets teams express many readiness decisions directly in the workflow. A training operation can depend on an upstream dataset-validation step in a DAG. A model-release operation can wait for manual approval. An evaluation branch can use conditional scheduling to skip work when its input conditions are false.

This keeps the reason for the wait alongside the operation and its dependencies. In a training-to-evaluation pipeline, use a dependency for required upstream work, an approval for a human decision, and a condition for a branch that should be skipped. These mechanisms avoid submitting useful work before its workflow prerequisites are resolved.

A Kubernetes scheduling gate complements that orchestration when a separate cluster controller needs an existing Pod to coordinate preparation or admission. That controller must create the gate and own its removal. Polyaxon workflow decisions and native gates operate at different stages, so identify which stage needs to wait and which component can prove readiness.

The Pending GPU guide helps distinguish an intentional wait from queueing, placement failures, and container startup problems.