Choose which failed Polyaxon jobs to retry

A worker that rejects an incompatible dataset will usually reject it again. A worker interrupted by a node disruption may finish on another node. Giving both failures the same retry policy spends attempts without making the job more recoverable.

Polyaxon exposes Kubernetes Pod failure policies through termination.podFailurePolicy. You can stop a Job on an application-specific exit code, exclude marked disruptions from its retry budget, and let other failures use a bounded number of retries. This guide puts those decisions into a small Polyaxon job, then explains what to inspect when the policy does not behave as expected.

Rules are checked from top to bottom; the first match determines the response.

Give a permanent error its own exit code

The useful distinction is whether another attempt can succeed with the same inputs and program. Decide that in the application before configuring the scheduler.

Our preflight worker accepts schema version 2. If the supplied configuration names another version, it prints the mismatch and exits with code 42. That code is our convention for this specific permanent error. It does not classify every exception, missing file, or network response as invalid input.

For a real training or evaluation worker, use the same pattern around a narrow validation boundary: a confirmed unsupported schema, an impossible parameter combination, or an incompatible checkpoint format. Leave ambiguous failures visible in the logs rather than assigning them a permanent meaning just to suppress retries.

The policy below handles three cases:

Putting the exit-code rule first makes the permanent-error evidence win if both conditions are present. Ignore preserves a counter; it does not turn failed work into a success.

Put the policy on a Polyaxon job

Use Polyaxon 2.18 or later for the simplified Polyaxonfile format below, with a compatible operator installation and a supported Kubernetes release that includes Pod failure policies. Kubernetes made the feature stable in 1.31. Polyaxon originally introduced its integration as Beta in the 2.13 release notes.

Save this as schema-preflight.yaml:

name: schema-preflight

container:
  image: python:3.12-slim

env:
  DATA_SCHEMA_VERSION: "1"

resources:
  cpu: "100m..500m"
  memory: "64Mi..128Mi"

environment:
  restartPolicy: Never

termination:
  maxRetries: 3
  timeout: 300
  podFailurePolicy:
    rules:
      - action: FailJob
        onExitCodes:
          containerName: polyaxon-main
          operator: In
          values: [42]
      - action: Ignore
        onPodConditions:
          - type: DisruptionTarget
            status: "True"

cmd: |
  python -u - <<'PY'
  import os
  import sys

  expected = "2"
  actual = os.environ["DATA_SCHEMA_VERSION"]
  if actual != expected:
      print(f"Unsupported schema version: expected {expected}, received {actual}")
      sys.exit(42)

  print("Schema preflight passed")
  PY

The example has no data download, external service, or training step. The image tag keeps it readable; use an approved immutable image digest for a maintained worker. CPU and memory ranges set requests and limits, as described in the simplified workload guide.

Submit it to your own project:

polyaxon run -p my-project -f schema-preflight.yaml

With version 1, the worker exits with 42, matching the permanent-error rule. Change the value to 2 and submit a new run: the program passes its preflight and exits normally.

The worker's generated container name is polyaxon-main. Restricting the rule to it prevents an unrelated helper container's exit code from acquiring the application's meaning. If you use a custom workload template, inspect the generated Pod and match the actual container name.

Bound replacements as well as counted failures

The termination specification maps maxRetries to the Job's backoffLimit and timeout to its active deadline. Keep restartPolicy: Never when using a Pod failure policy: the Job controller replaces failed Pods rather than restarting the worker inside the same Pod.

The example allows normal retry accounting for unclassified failures, while the five-minute deadline bounds the Job across replacement Pods. Repeated ignored disruptions can otherwise keep creating work without exhausting the counted-failure budget. The deadline applies even when budget remains.

DisruptionTarget is a condition Kubernetes sets for qualifying disruptions. A nonzero exit or a generic termination signal does not by itself establish that condition. Likewise, exit code 137 alone is not enough to identify an out-of-memory event; inspect the container's termination reason and surrounding evidence before adding a rule for it.

These rules act on failed Pods. They do not repair an invalid image reference or make an unschedulable Pod fit on a node. Use the relevant diagnostics for image pulls and GPU placement when the worker never starts.

Inspect the decision before changing the policy

Start with the run's logs and status in Polyaxon. For a policy mismatch, an administrator can inspect the underlying Job and failed Pod using their names from the cluster:

kubectl -n YOUR_NAMESPACE get job YOUR_JOB -o yaml
kubectl -n YOUR_NAMESPACE get pod YOUR_FAILED_POD -o yaml
kubectl -n YOUR_NAMESPACE describe job YOUR_JOB

Check that the rendered Job has the intended rule order, backoff limit, and active deadline. In its Pod template, check restartPolicy and the worker's container name. In the failed Pod, inspect containerStatuses, the exit code and termination reason, and any DisruptionTarget condition. These commands require cluster access; ordinary run review does not.

Do not infer which rule matched from a final failed status alone. Preserve the application message and Pod evidence before normal resource cleanup. The Kubernetes Job reference documents the matching and termination behavior; the native Kubernetes walkthrough goes deeper into the controller-level example.

Make a replacement useful

Once a policy permits another attempt, the application still has to recover. Write checkpoints to durable storage and restore the optimizer, scheduler, and data-progress state needed for the next update. Resume interrupted training covers that implementation.

Make output publication safe to repeat as well. An attempt may have written an object or completed an external request before its Pod failed. Use attempt-specific intermediate outputs and a checked completion record when the workflow has side effects.

Start with one exit code whose meaning your worker enforces, an explicit disruption rule, and bounded fallback retries. Review failures that remain unclassified before making the policy more aggressive.