Kubernetes learning projects for ML engineers

The fastest way to learn Kubernetes is to operate a small workload and explain every state transition. For an ML engineer, useful projects should connect Kubernetes primitives to familiar work: training, data, artifacts, scheduling, and model serving.

Use an isolated learning cluster and disposable data. Do not practice destructive commands or broad permissions in a shared production environment.

Build a safe learning environment

Follow the official Kubernetes tutorials and use a supported local or sandbox cluster. Create a dedicated namespace and make the context visible in every command.

kubectl --context learning-cluster create namespace ml-lab
kubectl --context learning-cluster --namespace ml-lab get all

Set CPU and memory limits so experiments cannot consume the host. Avoid real credentials, customer data, public load balancers, and privileged containers. Save manifests in version control and record what you expect before applying each change.

The projects below form a progression. Complete the diagnostic questions, not just the installation.

Project 1: run a finite training Job

Package a small training script in a container and run it as a Kubernetes Job. Declare resource requests, pass configuration through a ConfigMap, and write a result to durable storage.

Observe the Job, Pod, init, container, and termination states. Delete the Pod and explain how the Job controller reacts. Introduce an invalid image tag and identify why the application never starts.

This project teaches desired state, ownership, scheduling, image pulls, logs, exit reasons, and finite-work semantics.

Project 2: persist checkpoints

Mount a PVC and write periodic checkpoints. Stop the Pod, let the controller replace it, and resume from the retained data.

Then test a wrong mount path, insufficient capacity, and a volume topology conflict. Distinguish provisioning, attachment, mounting, permissions, throughput, and application-level file safety.

Compare PVC-backed storage with object storage for sharing artifacts across clusters. Polyaxon artifact connections show how durable outputs fit a tracked workflow.

Project 3: serve a model through a Service

Deploy a tiny model API with a Deployment and expose it through a ClusterIP Service. Add startup and readiness probes that represent actual model loading and serving state.

Roll out a new image digest and watch the Deployment create a new ReplicaSet. Break the Service selector and diagnose why the Service has no endpoints. Scale replicas and observe how traffic and readiness change.

This project connects Pods, Deployments, ReplicaSets, Services, probes, and controlled rollout.

Project 4: constrain scheduling

Label a node or use a simulated node pool, then apply node affinity, taints, tolerations, and topology spread to a Job. Create one configuration that can schedule and another that cannot.

Read the Pending Pod's scheduling events and explain every failed predicate. Change one constraint at a time rather than adding capacity immediately.

Use Polyaxon node scheduling to see how reusable platform policy avoids copying raw placement rules into each experiment.

Project 5: add observability

Collect Kubernetes object state, container resource metrics, application metrics, and structured logs. Create one dashboard that follows a Job from submission to completion.

Answer four separate questions:

• What resource capacity was requested?
• Was the workload admitted and scheduled?
• What resources did it use?
• Did it produce the expected result?

Add a trace around the model-serving request and link it to logs. Keep unique run identifiers out of unbounded metric labels.

Project 6: apply least privilege

Create a service account and a namespaced Role that grants only the API operations the workload needs. Disable automatic token mounting when it does not need the Kubernetes API.

Add a default-deny NetworkPolicy and permit only DNS plus the required artifact endpoint. Test an allowed and rejected connection from the workload identity.

Review the effective permissions and audit evidence. A successful deployment is not complete if it receives unnecessary cluster-wide access.

Project 7: rehearse failure and recovery

Create controlled failures: terminate a Pod, make a dependency slow, fill an ephemeral volume within its limit, revoke an unused credential, and temporarily make a node unavailable.

For each case, capture conditions, events, logs, metrics, and owner references before recovery. Write a short runbook with detection, diagnosis, safe action, and validation.

The goal is not chaos for its own sake. It is learning which controller or team owns each transition and which evidence disappears after a retry.

Turn projects into Polyaxon workflows

After understanding the Kubernetes objects, repeat the training, evaluation, and serving work through Polyaxon operations. Compare the raw manifests with the higher-level project, run, parameters, environment, logs, metrics, and artifacts.

Use the operations quick start and scheduling presets to make experiments reproducible. The Kubernetes layer should explain execution; Polyaxon should preserve the ML intent and evidence above it.

Complete these projects by predicting behavior, creating one controlled failure, and explaining the recovery. That practice builds operational understanding far faster than deploying a long list of sample applications without investigating how they work.