Helm charts for ML platforms

Helm packages Kubernetes resources as a versioned chart and renders templates with environment-specific values. It can make installing and upgrading an ML platform repeatable without duplicating large sets of manifests.

Helm does not replace infrastructure provisioning, application reconciliation, or environment design. A chart produces Kubernetes objects; operators still own the values, dependencies, policies, rollout, and recovery behavior around them.

Understand the chart structure

The official Helm chart documentation defines the main files:

Path	Purpose
`Chart.yaml`	Chart name, type, version, application version, and metadata
`values.yaml`	Default user-facing configuration
`templates/`	Kubernetes manifests rendered with values and release context
`charts/`	Packaged dependencies
`crds/`	CustomResourceDefinitions installed with special lifecycle behavior
`templates/NOTES.txt`	Optional post-install or upgrade guidance

Keep version separate from appVersion: the chart package and the installed application do not necessarily release together.

Treat the chart as an API. Values become a long-lived contract used by automation, operators, and downstream overrides.

Design values around intent

Expose decisions users genuinely need to control: image registry and digest, replica or worker counts, resource requests, storage, ingress, identity, scheduling, and approved integrations.

Avoid mirroring every Kubernetes field into values.yaml. Unlimited pass-through configuration makes validation difficult and weakens the chart's operating guarantees.

Use consistent nesting, types, and names. Prefer a small number of documented structures over many aliases. Provide safe defaults for development only when they cannot be mistaken for production policy.

Never place credentials in default values or commit them in environment value files. Reference Kubernetes Secrets or external secret mechanisms according to the platform's security model.

Render before applying

Inspect the complete manifest that values produce:

helm template polyaxon ./charts/polyaxon \
  --namespace polyaxon \
  --values environments/production.yaml

Review names, namespaces, image digests, resources, service accounts, security contexts, volumes, affinity, tolerations, and generated labels. Rendering catches template problems before they reach admission, but it does not prove the resources will schedule or the application will work.

Store the chart version, values, and rendered-diff review in the delivery record. Secret values should remain redacted.

Manage dependencies deliberately

A dependency can simplify packaging and also expand the chart's lifecycle. Pin compatible dependency versions and review their values, permissions, images, CRDs, and upgrade notes.

Avoid bundling databases, object stores, or observability systems merely because a chart exists. Production deployments may require independent ownership, backups, scaling, and failure domains.

Define whether a dependency is required, optional, externally provided, or managed separately. A chart upgrade should not unexpectedly replace a stateful service.

Handle CRDs separately

Helm installs CRDs from the crds/ directory before ordinary templates, but CRD upgrade and deletion require special care. A CRD is a cluster-wide API whose lifecycle can outlive one release.

Plan schema versions, controller compatibility, conversion, migration, and rollback explicitly. Do not assume uninstalling a chart should delete every custom resource and its data.

Our Kubernetes CRD guide covers API and controller lifecycle in more detail.

Design safe upgrades

Use helm upgrade --install for a consistent release workflow and choose atomic or wait behavior based on the resources involved:

helm upgrade --install polyaxon ./charts/polyaxon \
  --namespace polyaxon \
  --create-namespace \
  --values environments/production.yaml \
  --wait \
  --timeout 15m

Waiting for Kubernetes readiness does not validate user journeys, data migrations, or background controllers. Canary where possible and perform application-level checks after the release.

Rollback is not always safe when schemas, CRDs, storage, or external systems changed. Document forward-fix and restore procedures rather than relying on release history alone.

Secure templates and hooks

Review chart output as production code. Reject unnecessary privilege, host access, broad RBAC, default service-account tokens, mutable image tags, and externally reachable Services.

Hooks create Jobs or resources outside the ordinary install sequence. Make them idempotent, bounded by deadlines, observable, and safe to retry. Define cleanup so failed hooks do not accumulate credentials or block future releases.

Verify chart provenance and container-image provenance independently. A trusted chart can still reference an untrusted image, and a signed image can still be deployed with unsafe permissions.

Fit Helm into Polyaxon operations

Use Helm for installing and configuring Polyaxon platform components. Use Polyaxon operations for the repeatable execution of training, evaluation, data processing, and services on top of that platform.

Polyaxon platform observability supports the installed services, while scheduling presets centralize reusable workload policy without forcing every team to maintain chart overrides.

A reliable chart has a small, intentional configuration API; renders predictable Kubernetes objects; and has an owned upgrade and recovery path. Helm makes packaging easier, but operational clarity is what makes the package safe to run.