Install Polyaxon using kubeadm on Kubernetes

This guide explains how to install Polyaxon on an Arch Linux Kubernetes cluster: prepare a node with kubeadm, add networking and optional GPU support, configure shared NFS storage and TLS, then deploy and manage Polyaxon using its CLI or Helm. You can use one machine for a learning cluster or add worker nodes for a shared deployment.

Requirements

Prepare dedicated machines using the Arch Linux Kubernetes guide and the upstream kubeadm prerequisites. Install compatible kubeadm, kubelet, and kubectl versions, a CRI-compatible runtime such as containerd, and Helm 3. Install NFS client utilities on every node that will mount NFS volumes.

Arch provides the Kubernetes tools, containerd, and NFS utilities through its package repositories. On a new node, after checking the versions available for your chosen stack, install the prerequisites with:

sudo pacman -Syu kubeadm kubelet kubectl containerd ethtool nfs-utils

Install Helm 3 separately using the instructions below. The AUR *-bin packages, aurman, Docker 18.09, and Helm 2 versions in the original setup are no longer prerequisites. With kubeadm's default stacked-etcd topology, kubeadm runs etcd as a static Pod; an additional host etcd installation is unnecessary.

Record the versions before initialization:

kubeadm version
kubelet --version
kubectl version --client
containerd --version
helm version

Configure IP forwarding, swap behavior, kernel modules, firewall rules, and the runtime's cgroup driver according to the selected Kubernetes release and CNI. Reverse-path filtering depends on the network design; do not copy a global rp_filter setting without checking the CNI requirements. Avoid a partial Arch system upgrade that leaves Kubernetes, the kernel, and the runtime incompatible.

Add support for the NVIDIA runtime

For GPU nodes, follow the NVIDIA Container Toolkit installation guide for the selected runtime and drivers. Configure containerd when it is the Kubernetes runtime; editing Docker's daemon configuration does not configure containerd. Keep a copy of the existing runtime configuration and schedule its restart as a node-maintenance action.

Once the host driver and toolkit are installed, configure the NVIDIA runtime as containerd's default on each GPU node for the device-plugin setup below:

nvidia-smi
sudo nvidia-ctk runtime configure --runtime=containerd --set-as-default
sudo systemctl restart containerd

The Kubernetes device plugin is installed later; the runtime configuration alone does not advertise GPUs to the scheduler.

Start kubelet and the container runtime

For the containerd-based setup used here, enable the installed systemd services:

sudo systemctl daemon-reload
sudo systemctl enable --now containerd
sudo systemctl enable --now kubelet

The kubelet can restart while waiting for kubeadm to provide its configuration. Use its journal and kubeadm preflight output to diagnose failures rather than repeatedly reinitializing the node.

Configure some variables to save typing

Use explicit names for the infrastructure selected below. Replace the examples with your NFS server and, if exposing the gateway, a domain pointing to the ingress controller:

export POLYAXON_NFS_SERVER=nfs.example.com
export POLYAXON_DOMAIN=polyaxon.example.com
export POLYAXON_NODE_IP=192.168.10.2
export POLYAXON_POD_CIDR=10.244.0.0/16

POLYAXON_NODE_IP is the control-plane node address reachable by worker nodes. POLYAXON_POD_CIDR is the network reserved for Pods; use the same range in your CNI configuration and choose a different range if this example overlaps an existing network.

Configure Kubernetes

Initialize only a prepared node that is not already part of a cluster. kubeadm reset is a destructive recovery/decommissioning operation, not a routine prerequisite. Preserve an existing cluster's configuration and state before following a recovery procedure.

Basic configuration

For a single control-plane node using containerd and the Pod network selected above, initialize the cluster with:

sudo kubeadm init \
  --apiserver-advertise-address="$POLYAXON_NODE_IP" \
  --cri-socket=unix:///run/containerd/containerd.sock \
  --pod-network-cidr="$POLYAXON_POD_CIDR"

For a customized or multi-control-plane deployment, use a kubeadm configuration instead. Follow the cluster creation guide and record the API endpoint, CRI socket, Kubernetes version, and Pod/Service networks. Generate a starting file with the installed kubeadm version:

kubeadm config print init-defaults > kubeadm-config.yaml

Replace the generated address and node defaults, set networking.podSubnet to the chosen Pod CIDR, and review the configuration. Then use this command instead of the direct initialization command above:

sudo kubeadm init --config kubeadm-config.yaml

Use the generated administrator kubeconfig with kubectl. Keep it separate from other cluster configurations and restrict its permissions:

mkdir -p "$HOME/.kube"
sudo install -m 600 -o "$(id -u)" -g "$(id -g)" \
  /etc/kubernetes/admin.conf "$HOME/.kube/polyaxon-kubeadm.conf"
export KUBECONFIG="$HOME/.kube/polyaxon-kubeadm.conf"
kubectl config current-context

You might want to add KUBECONFIG to your rc file.

Watch the control-plane Pods start with kubectl get pods --all-namespaces. CoreDNS can remain Pending until the CNI is installed.

Network configuration

The original setup used Canal, which combines Flannel networking with Calico network policy. You can keep that design using the current Canal installation instructions. Select a compatible release from Calico's releases, replace the placeholder with its tag including the v prefix, and download its Kubernetes API datastore manifest:

export CALICO_VERSION=REPLACE_WITH_CALICO_RELEASE_TAG
curl -fL -o canal.yaml \
  "https://raw.githubusercontent.com/projectcalico/calico/$CALICO_VERSION/manifests/canal.yaml"

Review canal.yaml. If POLYAXON_POD_CIDR differs from the manifest's 10.244.0.0/16 default, replace that range with your chosen Pod CIDR before applying it:

kubectl apply -f canal.yaml

If you choose another CNI, follow that project's installation procedure instead. Keep its version and values with the kubeadm configuration.

Inspect node readiness and DNS after the CNI is installed:

kubectl get nodes
kubectl --namespace kube-system get pods

Continue only after the nodes are Ready and cluster DNS is available. Join additional prepared worker nodes using kubeadm's join procedure and verify their runtime, networking, and storage access too.

Create the namespace used by the chart, its storage claims, and any optional certificate resources:

kubectl create namespace polyaxon

Allow workloads on a single-node learning cluster

For a single-node learning cluster, remove the control-plane scheduling taint from that specific node after checking its name. Replace CONTROL_PLANE_NODE below. Keep the taint on production control-plane nodes and use worker nodes for ML workloads.

kubectl taint node CONTROL_PLANE_NODE node-role.kubernetes.io/control-plane:NoSchedule-

Install Helm

Install Helm 3 using the Helm installation instructions. It uses your kubeconfig directly and does not require Tiller or a cluster-admin binding for Tiller. The Helm 2 setup from the original environment is not part of this installation.

Configure Helm repositories

Add the Polyaxon repository and inspect available chart versions:

helm repo add polyaxon https://charts.polyaxon.com
helm repo update
helm search repo polyaxon/polyaxon --versions

Add NVIDIA capabilities

After configuring the host driver and NVIDIA runtime on the GPU nodes, install the NVIDIA device plugin with Helm. If GPU Operator already manages the plugin, use that installation instead of deploying a second copy.

Add NVIDIA's chart repository and list the available versions:

helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update
helm search repo nvdp/nvidia-device-plugin --versions

Replace the chart-version placeholder with a compatible version from that list, without a v prefix, and review its values:

export NVIDIA_DEVICE_PLUGIN_VERSION=REPLACE_WITH_CHART_VERSION
helm show values nvdp/nvidia-device-plugin --version "$NVIDIA_DEVICE_PLUGIN_VERSION"

The chart's default node affinity recognizes GPU labels. If Node Feature Discovery has not already labeled your GPU nodes, replace GPU_NODE and label each node whose driver and runtime you configured:

kubectl label node GPU_NODE nvidia.com/gpu.present=true

Install the selected chart and inspect its DaemonSet and Pods:

helm upgrade --install nvdp nvdp/nvidia-device-plugin \
  --namespace nvidia-device-plugin --create-namespace \
  --version "$NVIDIA_DEVICE_PLUGIN_VERSION"
kubectl get daemonsets,pods --namespace nvidia-device-plugin

Inspect the node's advertised resources before submitting GPU work:

kubectl describe nodes

Check the resource names and quantities against the configured GPU mode. Whole GPUs, MIG profiles, and shared resources expose different contracts; see MIG versus time-slicing.

Configure ingress access

For local access, keep the Polyaxon gateway internal and use polyaxon port-forward; ingress and public certificates are optional. For a shared endpoint, choose and install a maintained controller and configure its network exposure explicitly. The community Ingress NGINX project retired in March 2026, so the old mandatory.yaml installation is not carried forward here.

Record the controller's IngressClass, reachable address, TLS behavior, and access policy. Verify DNS and firewall rules against that address. A Service's externalIPs field does not provision or route an external IP for you.

kubectl get ingressclass
kubectl get services --all-namespaces

For a concrete Ingress-based setup, you can install Traefik with its Helm chart. Skip this installation if you already have a suitable controller. Select a chart version compatible with the cluster:

helm repo add traefik https://traefik.github.io/charts
helm repo update
helm search repo traefik/traefik --versions
export POLYAXON_INGRESS_CHART_VERSION=REPLACE_WITH_TRAEFIK_CHART_VERSION
helm show values traefik/traefik --version "$POLYAXON_INGRESS_CHART_VERSION"

Save these overrides as ingress-controller.yaml. They use the chart's IngressClass and NodePort settings; change the example NodePorts if they are already allocated:

ingressClass:
  enabled: true
  isDefaultClass: false
  name: polyaxon-traefik
providers:
  kubernetesIngress:
    enabled: true
  kubernetesGateway:
    enabled: false
service:
  type: NodePort
ports:
  web:
    nodePort: 30080
  websecure:
    nodePort: 30443

Install and inspect the controller:

helm install polyaxon-ingress traefik/traefik \
  --namespace polyaxon-ingress --create-namespace \
  --version "$POLYAXON_INGRESS_CHART_VERSION" \
  -f ingress-controller.yaml --wait --timeout 10m
kubectl get pods,services --namespace polyaxon-ingress
kubectl get ingressclass polyaxon-traefik

Use polyaxon-traefik for the IngressClass placeholders in the Issuer and Polyaxon configuration below. Point the domain at your routed address and configure your router or load balancer to forward public TCP port 80 to a reachable node's port 30080 and port 443 to 30443. NodePort allocation alone does not create those routes. If your cluster provides a LoadBalancer service implementation, you can use that service type and its assigned address instead.

The optional HTTP-01 certificate example below requires an Ingress-compatible controller and public access to the challenge path on port 80. For private domains or a Gateway API deployment, select the corresponding cert-manager solver and routing configuration instead.

Install cert-manager

Use cert-manager's Helm installation to manage certificates if it is not already installed. Choose a supported version, replace the placeholder, and install its CRDs together with the release:

export CERT_MANAGER_VERSION=REPLACE_WITH_CERT_MANAGER_VERSION
helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --namespace cert-manager --create-namespace --version "$CERT_MANAGER_VERSION" \
  --set crds.enabled=true --wait --timeout 10m

Inspect kubectl get pods --namespace cert-manager and resolve webhook readiness before creating Issuers.

Configure the Issuer for polyaxon

For the optional TLS setup, save the following as issuer.yaml in the namespace created earlier. Replace the email and REPLACE_WITH_INGRESS_CLASS with an installed class supported by your controller. This uses the ACME HTTP-01 solver and the staging service so you can establish the challenge path before requesting a publicly trusted certificate:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-staging
  namespace: polyaxon
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging-account
    solvers:
      - http01:
          ingress:
            ingressClassName: REPLACE_WITH_INGRESS_CLASS

kubectl apply -f issuer.yaml
kubectl describe issuer letsencrypt-staging --namespace polyaxon

An Issuer being Ready does not prove that an individual Certificate has been issued. Inspect its Certificate, Order, and Challenge resources too. After successful staging issuance, create a separate production Issuer using https://acme-v02.api.letsencrypt.org/directory and update the requested certificate. Staging certificates are not trusted by browsers; do not disable TLS verification to treat them as production certificates.

Using local directories for all nodes of polyaxon

Configure the nfs directory:

We can add nfs directories to our configuration in case we want to share those directories with all the cluster.

Arch Linux NFS setup.

Our nfs directories are /srv/nfs/data/ and /srv/nfs/outputs.

Create the directories on the NFS server, set ownership for the intended workload UID/GID, and restrict exports and the firewall to the actual client nodes. For example, if those nodes belong to the private 192.168.10.0/24 subnet, /etc/exports could contain:

/srv/nfs/data       192.168.10.0/24(rw,sync,root_squash,no_subtree_check)
/srv/nfs/outputs    192.168.10.0/24(rw,sync,root_squash,no_subtree_check)

Replace that example subnet; loopback addresses do not grant access to remote worker nodes. Reload the configured exports with sudo exportfs -ra and inspect them with sudo exportfs -v.

Start and enable the service:

sudo systemctl enable nfs-server.service
sudo systemctl start nfs-server.service

Create persistence volumes for our nfs directories

Adjust the capacities to the available exports. The following binds each claim to a named static PV, avoiding accidental provisioning through a default StorageClass. Kubernetes capacity declarations do not create NFS quotas; enforce capacity and backups on the server.

cat <<EOF > /tmp/pvcdata.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: polyaxon-pv-data
spec:
  capacity:
    storage: 3Ti
  storageClassName: ""
  persistentVolumeReclaimPolicy: Retain
  accessModes:
    - ReadWriteMany
  nfs:
    path: /srv/nfs/data
    server: $POLYAXON_NFS_SERVER
    readOnly: false
  claimRef:
    namespace: polyaxon
    name: polyaxon-pvc-data
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name:  polyaxon-pvc-data
  namespace: polyaxon
spec:
  storageClassName: ""
  volumeName: polyaxon-pv-data
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 3Ti
EOF
kubectl create --namespace=polyaxon -f /tmp/pvcdata.yaml

cat <<EOF > /tmp/pvcoutputs.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: polyaxon-pv-outputs
spec:
  capacity:
    storage: 256Gi
  storageClassName: ""
  persistentVolumeReclaimPolicy: Retain
  accessModes:
    - ReadWriteMany
  nfs:
    path: /srv/nfs/outputs
    server: $POLYAXON_NFS_SERVER
    readOnly: false
  claimRef:
    namespace: polyaxon
    name: polyaxon-pvc-outputs
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name:  polyaxon-pvc-outputs
  namespace: polyaxon
spec:
  storageClassName: ""
  volumeName: polyaxon-pv-outputs
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 256Gi
EOF
kubectl create --namespace=polyaxon -f /tmp/pvcoutputs.yaml

Inspect the claims and their bound volumes:

kubectl --namespace polyaxon get pvc polyaxon-pvc-data polyaxon-pvc-outputs
kubectl get pv polyaxon-pv-data polyaxon-pv-outputs

Binding does not prove that every node can mount and write the export. Verify client utilities, network access, and workload permissions before relying on it for artifacts. A single NFS server remains a failure domain; Retain is not a backup.

Install Polyaxon

Install the Polyaxon CLI in a virtual environment so it is available for deployment and later client operations:

python3 -m venv .venv-polyaxon
source .venv-polyaxon/bin/activate
python -m pip install polyaxon

Use the Polyaxon platform installation guide and the selected chart's values for edition-specific services and database persistence. Select a chart version from the Helm repository output and replace the placeholder below. Both deployment methods will use this version, release name, namespace, and configuration file:

export POLYAXON_CHART_VERSION=REPLACE_WITH_CHART_VERSION
helm show values polyaxon/polyaxon --version "$POLYAXON_CHART_VERSION"

This starting configuration keeps the gateway internal and registers the NFS claims using the connection schema. Retain config.yaml for future upgrades:

cat <<EOF > config.yaml
deploymentType: kubernetes
deploymentVersion: "$POLYAXON_CHART_VERSION"
releaseName: polyaxon
namespace: polyaxon
rbac:
  enabled: true
gateway:
  service:
    type: ClusterIP
ingress:
  enabled: false
connections:
  - name: nfs-data
    kind: volume_claim
    schema:
      volumeClaim: polyaxon-pvc-data
      mountPath: /data
artifactsStore:
  name: nfs-artifacts
  kind: volume_claim
  schema:
    volumeClaim: polyaxon-pvc-outputs
    mountPath: /artifacts
EOF

For the shared HTTPS endpoint, replace the ingress section in config.yaml with the following, using the domain from POLYAXON_DOMAIN and your installed IngressClass. This connects the chart's Ingress to the cert-manager Issuer created earlier:

ingress:
  enabled: true
  className: REPLACE_WITH_INGRESS_CLASS
  hostName: polyaxon.example.com
  annotations:
    cert-manager.io/issuer: letsencrypt-staging
  tls:
    - secretName: polyaxon-letsencrypt
      hosts:
        - polyaxon.example.com

After verifying staging issuance, create the production Issuer described above and change the annotation to letsencrypt-prod. The Ingress, Issuer, and TLS Secret belong in the polyaxon namespace. If you already have a valid TLS Secret, reference it here and omit the cert-manager annotation. Configure the edition's authentication and network access policy before opening the endpoint to other users.

For an edition that provisions a root user, retain the initial account settings in the same deployment configuration:

user:
  username: rootuser
  email: REPLACE_WITH_ADMIN_EMAIL
  password: REPLACE_WITH_A_UNIQUE_PASSWORD

Replace these placeholders before deploying and keep any configuration containing credentials private. These account settings do not add authentication to Community Edition; keep its endpoint within the intended trusted access boundary.

Deploy with the Polyaxon CLI

The admin commands use Helm to manage the deployment. Check the configuration and required tools, then inspect a dry run:

polyaxon admin deploy -f config.yaml --check
polyaxon admin deploy -f config.yaml --dry-run

When the configuration is ready, deploy it:

polyaxon admin deploy -f config.yaml

Deploy with Helm

You can use Helm directly for the same deployment. Choose this method or the CLI method above:

helm template polyaxon polyaxon/polyaxon --namespace polyaxon \
  --version "$POLYAXON_CHART_VERSION" -f config.yaml
helm install polyaxon polyaxon/polyaxon --namespace polyaxon \
  --version "$POLYAXON_CHART_VERSION" -f config.yaml --wait --timeout 10m

Inspect the rendered configuration before installation. Once deployed, inspect readiness with kubectl get pods,pvc,services --namespace polyaxon and follow the chart's access instructions. For the HTTPS option, inspect kubectl get ingress,certificate --namespace polyaxon too.

Upgrade Polyaxon or change its configuration

Edit the saved config.yaml when changing the deployment. To upgrade to a newer chart, update both deploymentVersion in that file and POLYAXON_CHART_VERSION to the selected version. Review the release notes, migration requirements, and backups first.

With the Polyaxon CLI:

polyaxon admin upgrade -f config.yaml --check
polyaxon admin upgrade -f config.yaml --dry-run
polyaxon admin upgrade -f config.yaml

Or with Helm:

helm upgrade polyaxon polyaxon/polyaxon --namespace polyaxon \
  --version "$POLYAXON_CHART_VERSION" -f config.yaml --wait --timeout 10m

Inspect the Pods and persistent claims after the upgrade, then verify that you can access the dashboard and run an operation using the existing data connection.

Configure the client and access Polyaxon

For the internal gateway, activate the CLI environment and start port forwarding:

source .venv-polyaxon/bin/activate
polyaxon port-forward

Keep the forwarding process running and open the localhost URL it prints. The command also configures the CLI for that local endpoint. Run subsequent CLI commands in another terminal with the same virtual environment activated.

For the shared domain, configure the HTTPS endpoint after its production certificate is Ready:

polyaxon config set --host="https://$POLYAXON_DOMAIN" --verify-ssl=true

For an authenticated deployment, log in with the username configured for that installation. Enter the password when prompted:

polyaxon login --username=YOUR_USERNAME
polyaxon project ls

Complete the setup with a small Polyaxon operation that uses the intended data connection and produces a retrievable artifact. Preserve the Kubernetes, CNI, runtime, GPU, chart, and storage configurations with that result so the installation can be reproduced and recovered.

Tear down Polyaxon

When you want to remove the deployment, first stop or finish running operations and preserve the database and artifacts you need. Use the same configuration file to identify the release and namespace:

polyaxon admin teardown -f config.yaml

The CLI asks whether to execute pre-delete hooks. Enable them when you want the chart's cleanup hooks to run; skipping hooks requires handling any remaining operation resources yourself.

The Helm alternative is:

helm uninstall polyaxon --namespace polyaxon

The NFS PVs and PVCs created explicitly in this guide are separate from the Helm release. Keep them and their server exports if you plan to reinstall Polyaxon against the same data. Review chart-managed database storage separately. Removing Polyaxon does not remove kubeadm, the CNI, cert-manager, the GPU integration, or the NFS server; those can continue serving other workloads.