Retry failed evaluation shards with Indexed Jobs

An evaluation is split into many independent shards. Most finish, one repeatedly encounters a service failure, and another cannot read its assigned data. Restarting the whole evaluation wastes completed work. Treating the surviving results as a complete evaluation hides missing evidence.

Kubernetes Indexed Jobs give each shard a stable completion index. Per-index retry budgets, stable since Kubernetes 1.33, let one failing shard exhaust its budget without immediately abandoning every other shard. The final Job can still fail when any index fails. Job controller documentation.

This is the Kubernetes execution layer beneath the broader batch evaluation contract: stable inputs, durable outputs, and an explicit completeness check.

Map an index to immutable work

An Indexed Job numbers its required completions from zero. The container receives its index in JOB_COMPLETION_INDEX. The application must map that number to a stable assignment. Indexed processing guide.

For example, a versioned manifest might assign each index a list of case IDs. Keep the manifest unchanged across retries. Recomputing assignments from a mutable directory listing can make “retry shard 4” evaluate different cases from its first attempt.

The index also does not provide exactly-once execution. A replacement or duplicate attempt must be safe to run. Use a durable result identity containing the evaluation revision, candidate, case, and repetition; retain an attempt identifier separately. An object-store write protocol or database constraint must resolve duplicates deliberately.

Exercise the retry decision with a small fixture

This disposable example needs Kubernetes 1.33 or later, permission to create a namespace and Jobs, and access to the Python image. It runs no model and produces no evaluation scores. Its purpose is to demonstrate six indexes with two distinct failure classes.

Save it as indexed-eval-demo.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: indexed-eval-demo
---
apiVersion: batch/v1
kind: Job
metadata:
  name: evaluation
  namespace: indexed-eval-demo
spec:
  completionMode: Indexed
  completions: 6
  parallelism: 2
  backoffLimitPerIndex: 1
  maxFailedIndexes: 2
  activeDeadlineSeconds: 600
  podFailurePolicy:
    rules:
      - action: FailIndex
        onExitCodes:
          containerName: worker
          operator: In
          values: [42]
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: worker
          image: python:3.12-slim
          command:
            - python
            - -c
            - |
              import json
              import os
              import sys

              index = int(os.environ["JOB_COMPLETION_INDEX"])
              exit_code = 42 if index == 2 else 1 if index == 4 else 0
              print(json.dumps({
                  "fixture": "indexed-retry-demo",
                  "index": index,
                  "exit_code": exit_code,
              }), flush=True)
              sys.exit(exit_code)
          resources:
            requests:
              cpu: "100m"
              memory: "64Mi"
            limits:
              cpu: "500m"
              memory: "128Mi"

The Python tag is convenient for a disposable example; use an approved digest for a reproducible environment. Exit code 42 is an application convention defined by this fixture, not a universal signal for invalid data.

The FailIndex rule stops retries for index 2. Index 4 always exits with code 1, so its ordinary per-index budget allows one retry before failure. This follows the supported combination of Pod failure policy and per-index backoff.

Inspect completion and failure separately

Create the example and inspect it while it runs:

kubectl apply -f indexed-eval-demo.yaml
kubectl get job evaluation -n indexed-eval-demo -o yaml
kubectl get pods -n indexed-eval-demo -l batch.kubernetes.io/job-name=evaluation

After controller reconciliation, and assuming no unrelated infrastructure failures or deadline expiry, the expected outcome is:

These are expected results, not captured execution output. Inspect the actual Job conditions and Pod logs to explain any difference.

maxFailedIndexes: 2 aborts remaining work when the failed-index count exceeds two; it does not declare success when two shards fail. With this fixture, the other indexes can finish, and the Job ultimately fails because failed indexes remain. The deadline is an independent bound on the whole Job.

Reconcile results before reporting a score

A real evaluator should write durable shard receipts containing its manifest revision, assigned cases, completed cases, failures, and output locations. Reconcile those receipts against the expected manifest before calculating the release report.

Do not turn a failed or absent shard into a zero score, and do not silently exclude it from the denominator. Distinguish a model's response failure from infrastructure that prevented evaluation. The candidate completeness guide develops that reporting boundary.

Once you have inspected the fixture, remove only its disposable namespace:

kubectl delete namespace indexed-eval-demo

Choose the orchestration boundary in Polyaxon

Polyaxon matrix operations offer a practical way to make every evaluation shard an individually tracked run. Define the shard IDs as discrete input values, set a concurrency limit, and reuse the evaluator component. Each shard can retain its own metrics, logs, and result artifacts, making a missing or failed shard visible in the evaluation workflow.

For the six-shard example, record the expected IDs and each shard's completed item count, dataset version, and result location. A downstream aggregation step should reconcile those records before reporting a score. Polyaxon's artifact logging keeps the shard reports and reconciliation report available for review; the aggregation code still owns the completeness check.

A native Indexed Job is another useful execution choice when one Kubernetes controller should manage the indexed completions. In that integration, connect the Job's final status and durable outputs to the Polyaxon workflow, and decide which layer owns retries. A Polyaxon matrix creates separate operations; it does not implicitly turn them into an Indexed Job. Start with the six-index fixture to establish failure behavior before adding the real evaluator.