Use Kubernetes labels and annotations for ML

Kubernetes metadata connects resources that are otherwise independent API objects. Labels identify and select sets of objects. Selectors turn those labels into controller, Service, policy, and query behavior. Annotations store non-identifying context that tools can read without making it part of selection.

For ML platforms, a deliberate metadata contract makes workloads searchable and attributable without placing every experiment detail into the Kubernetes API.

Use labels for bounded identity

Labels are key-value pairs intended for identifying attributes. They support efficient list and watch operations, so values must be short and labels must stay within Kubernetes syntax limits.

Good platform labels describe stable, bounded dimensions:

metadata:
  labels:
    app.kubernetes.io/name: trainer
    app.kubernetes.io/component: worker
    app.kubernetes.io/part-of: fraud-model
    app.kubernetes.io/managed-by: polyaxon
    platform.example.com/team: risk-ml
    platform.example.com/environment: production

Use your organization's DNS prefix for labels written by automation. The kubernetes.io/ and k8s.io/ prefixes are reserved for Kubernetes components. The official recommended labels provide interoperable application keys.

Do not put secrets, free-form prompts, dataset URLs, timestamps, or arbitrary parameter values in labels.

Treat selectors as behavior

A selector matches objects by labels. Deployments use selectors to decide which Pods they own; Services use them to choose endpoints; policies and operational tools use them to define scope.

A selector is therefore not decorative metadata. An overly broad Service selector can send traffic to the wrong model version. Overlapping controller selectors can create conflicting ownership. A policy selector that misses a workload can remove the intended protection.

Inspect a set before acting on it:

kubectl get pods \
  --context acme-production \
  --namespace ml-team \
  --selector 'app.kubernetes.io/part-of=fraud-model,platform.example.com/environment=production' \
  --show-labels

Comma-separated requirements are logical AND. Set-based selectors can match several values, but selector capabilities differ between API fields. The labels and selectors documentation describes equality- and set-based forms.

Keep controller labels consistent

For a Deployment, .spec.selector.matchLabels must match labels in .spec.template.metadata.labels. Treat controller selector labels as part of the resource's identity. Many selector fields are immutable after creation because changing them could orphan or adopt Pods unexpectedly.

Separate identity labels from mutable descriptive labels. For example, a model service's application and instance can participate in its selector, while a release channel or review status can be changed without redefining ownership—provided no controller or policy relies on it.

Document every label used by a Service, controller, NetworkPolicy, PodDisruptionBudget, cost report, or admission rule before modifying its meaning.

Use annotations for non-identifying context

Annotations are also key-value metadata, but they are not used by Kubernetes selectors. They can hold larger or structured strings for build provenance, links, checksums, contact information, or tool configuration.

Useful ML annotations include:

• source revision and immutable image digest;
• a Polyaxon operation or project URL;
• model, data, or artifact lineage references;
• the checksum that should trigger a configuration rollout;
• an owner or runbook link;
• a policy decision or admission audit reference.

Annotations are not a secret store. Anyone able to read the object can read its annotations. Keep credentials and tokens in an approved secret-management path.

Avoid telemetry-cardinality leaks

Kubernetes labels and Prometheus labels share a name but not an automatic safety guarantee. Exporters may turn Kubernetes metadata into metric labels. A unique run ID, commit, Pod UID, model version, and dataset revision multiplied across several metrics can create millions of time series.

Define an allowlist for metadata exported to metrics. Keep unique identifiers in logs, traces, or linked operation metadata, where they can be searched without multiplying every metric series. Measure the cardinality impact before adding a new dimension.

Design an ML metadata contract

A useful contract specifies:

Keep keys versioned and platform-owned. Validate required labels at admission, but avoid letting schema changes block emergency recovery without a documented path.

Let Polyaxon preserve experiment context

Polyaxon connects generated Kubernetes resources to the project and operation that own them. Use Kubernetes metadata for the subset needed by scheduling, policy, inventory, and troubleshooting. Keep rich experiment parameters, metrics, artifacts, lineage, and status in Polyaxon.

That division makes both systems more reliable: Kubernetes receives compact operational identity, while ML users retain the detailed context needed to compare and reproduce work.