Kubernetes CronJobs for ML automation

A Kubernetes CronJob creates Jobs on a recurring schedule. It is suitable for bounded, repeatable work such as refreshing a dataset manifest, running a validation suite, cleaning an approved cache, or launching a periodic report.

It is not an exactly-once transaction system. Schedules can be missed or duplicated under some conditions, and a Job can fail after producing partial external effects. Reliable automation begins with idempotent work and explicit concurrency and deadline policy.

Understand the controller chain

The CronJob controller evaluates the schedule and creates a Job. The Job controller creates and tracks Pods. The Kubernetes CronJob documentation recommends making Jobs idempotent because scheduling is approximate.

This separation gives each scheduled execution its own status and Pods, but it also means three layers can affect an outcome:

• CronJob schedule and concurrency policy;
• Job retry and completion policy;
• Pod scheduling, execution, and termination.

Monitor the final work outcome, not only whether a Job object was created.

Define the schedule completely

Use batch/v1, quote the cron expression, and set the time zone explicitly:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-evaluation
  namespace: ml-team
spec:
  schedule: "0 2 * * *"
  timeZone: "Etc/UTC"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 1800
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 2
      ttlSecondsAfterFinished: 604800
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: evaluator
              image: registry.example.com/evaluator@sha256:REPLACE_WITH_DIGEST
              args: ["--dataset", "evaluation-current"]

Replace the placeholders with approved immutable inputs. .spec.timeZone uses an IANA time-zone name. Prefer UTC for infrastructure schedules unless a business calendar genuinely requires local time and daylight-saving behavior.

Choose concurrency by business semantics

concurrencyPolicy controls overlap among Jobs created by the same CronJob:

Forbid does not queue every missed occurrence for later. Replace does not make partial external writes disappear. Design the application around the chosen semantics.

For expensive training or evaluation, platform queues may provide better admission and prioritization than unconstrained CronJob Pod creation.

Bound late starts

startingDeadlineSeconds limits how late the controller may start a missed schedule. Choose it from the value of the work. A report that is useful for several hours may tolerate delay; a refresh superseded by the next interval may not.

Avoid an extremely small deadline. Control-plane reconciliation is not a real-time scheduler, and a value below the controller's practical check interval can make runs unreliable.

The deadline controls creation time, not execution duration. Bound workload runtime separately with Job and application-level timeouts.

Make the work idempotent

Assign each logical occurrence a durable business key, such as dataset plus scheduled interval. Before writing externally, check whether that key already completed. Write outputs to a temporary version, validate them, then publish atomically where the storage system supports it.

Retries and duplicate Jobs should converge on one valid outcome. Side effects such as emails, model promotions, or record mutation need deduplication at the destination; a Kubernetes object name is not a cross-system transaction.

Pause and inspect safely

Suspend future schedules without deleting history:

kubectl --context production --namespace ml-team patch cronjob nightly-evaluation \
  --type=merge \
  --patch '{"spec":{"suspend":true}}'

Inspect the CronJob and its Jobs:

kubectl --context production --namespace ml-team describe cronjob nightly-evaluation
kubectl --context production --namespace ml-team get jobs \
  --selector app=nightly-evaluation

Label the Job template deliberately if you plan to query its Jobs by label. Do not infer success from the CronJob's last schedule time; inspect the Job condition and the application result.

Use Polyaxon for ML-aware scheduling

Kubernetes CronJobs work well for cluster-local operational tasks. ML automation often also needs run lineage, parameters, artifacts, queues, approvals, conditional dependencies, and comparison across executions. Polyaxon's pipelines and automation add that context while Kubernetes remains the execution substrate.

Whichever scheduler owns the trigger, keep one source of truth. Record the schedule, time zone, input versions, concurrency semantics, timeout, retry policy, and result. Reliable recurring work is a series of explainable executions—not merely a cron expression that exists in the cluster.