Kubernetes Workloads Basics

Overview

Kubernetes provides the orchestration layer for Polyaxon workloads submitted to a Kubernetes cluster. Understanding its basic resources helps you diagnose a pending training job, inspect a failed container, or understand how a notebook is exposed.

The standalone examples below require a configured kubectl context and permission to create resources. Save each manifest under the filename used by its command. Check the target cluster, then create the disposable namespace used throughout:

kubectl config current-context
kubectl create namespace workloads-demo

Pin approved image digests for long-lived workloads; the image tags here are examples rather than a version support policy.

Polyaxon operations

When a user submits a job or service to the Kubernetes runtime, Polyaxon creates an operation custom resource. Its CustomResourceDefinition (CRD) defines the resource's API schema; the Polyaxon operator watches these resources, manages the underlying workloads, and reports status transitions to Polyaxon's API.

The following schema shows how a Polyaxon operation is constructed and submitted to Kubernetes:

• Operation with job runtime

Operation
└──Job
   └──Pod
      └──Containers

• Operation with service runtime

Operation
├──Deployment
│  └──ReplicaSet
│     └──Pods
│        └──Containers
└──Service (selects the Pods by label)

• Operation with TFJob runtime

Operation
└──TFJob
   └──Pods
      └──Containers

Other distributed runtimes follow a similar pattern with their corresponding workload controllers. The required CRDs and controllers must be installed for the runtime you select. A Service routes traffic to Pods; it does not own or create a Deployment.

Pods

Pods are the smallest deployable units of computing that you can create and manage in Kubernetes. When using Polyaxon, pods are created automatically for each job or service.

Creating a Pod

Users can create a pod directly using the Kubernetes API, or by using a configuration file like the following:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:
  containers:
    - name: busybox
      image: busybox:stable
      command: [ 'sh', '-c', 'echo The app is running! && sleep 3600' ]

To create the Pod shown above, run the following command:

kubectl apply --namespace workloads-demo -f pod.yaml
kubectl wait --namespace workloads-demo --for=condition=Ready pod/busybox --timeout=120s

Executing a command in a Pod

To execute a command in a Pod, use the kubectl exec command. For example, to execute the command sh in the Pod busybox:

kubectl exec --namespace workloads-demo -it busybox -- sh

Another way to execute a command in a Pod is to use the kubectl run command. For example, to execute the command echo hello in a new Pod running the image busybox:

kubectl run hello --namespace workloads-demo --image=busybox:stable --restart=Never --command -- echo hello
kubectl logs --namespace workloads-demo --pod-running-timeout=120s hello

For pods managed by a Polyaxon workload, the user can use the Polyaxon CLI or UI to execute commands without having to learn about the underlying Kubernetes concepts.

Show logs for a Pod

To show the logs for a Pod, use the kubectl logs command. For example, to show the logs for the Pod busybox:

kubectl logs --namespace workloads-demo busybox

To show the logs and follow the output, use the -f flag. For example, to show the logs for the Pod busybox and follow the output:

kubectl logs --namespace workloads-demo -f busybox

For pods managed by a Polyaxon workload, the user can use the Polyaxon CLI or UI to stream logs without having to learn about the underlying Kubernetes concepts.

Notes

• Pods are generally not created directly and are created using workload resources. However, it's useful to understand the Pod structure, and sometimes it is useful to create a Pod directly for debugging reasons.
• A Pod's restart policy controls container restarts within that Pod. A workload controller such as a Deployment or Job can create replacement Pods; a standalone Pod deleted from the API is not automatically recreated.
• Users interacting with Polyaxon's API will never need to create pods directly, it's also important to note that pods created manually will not be controlled and managed by Polyaxon.
• Cleanup of completed Polyaxon workloads depends on the configured termination and retention policies. Preserve logs and artifacts before removing resources you still need for diagnosis.

Jobs

A job creates one or more pods and ensures that a specified number of them successfully terminate. Most of the machine learning workload is executed as a job, and Polyaxon provides a few job types that are supported by the platform. Jobs come with a few extra features, such as retries, backoff, and timeout, which are exposed by Polyaxon's interface.

Creating a Job

A user can create a job directly using the Kubernetes API, or by using a configuration file like the following:

apiVersion: batch/v1
kind: Job
metadata:
  name: pi
spec:
  template:
    spec:
      containers:
        - name: pi
          image: perl:5.40
          command: [ "perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)" ]
      restartPolicy: Never
  backoffLimit: 4

To create the Job shown above, run the following command:

kubectl apply --namespace workloads-demo -f job.yaml
kubectl wait --namespace workloads-demo --for=condition=Complete job/pi --timeout=120s
kubectl logs --namespace workloads-demo job/pi

The logs should contain the computed digits of pi. restartPolicy belongs under the Pod template's spec and must be Never or OnFailure for a Job. With Never, a failed container terminates its Pod and the Job controller can create another Pod within the configured backoff limit. See the Kubernetes Job documentation for retry and completion behavior.

Notes

Users interacting with Polyaxon's API will never need to create jobs directly, it's also important to note that jobs created manually will not be controlled and managed by Polyaxon.

Services

A service is a named abstraction that groups pods and provides a single point of entry for accessing them. Services are used to expose notebooks, tensorboards, and other services.

Creating a Service

A user can create a service directly using the Kubernetes API, or by using a configuration file like the following:

apiVersion: v1
kind: Service
metadata:
  name: tensorboard
spec:
  selector:
    app: tensorboard
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 6006
      targetPort: 6006

To create the Service shown above, run the following command:

kubectl apply --namespace workloads-demo -f service.yaml

Deployments

A Service needs reachable endpoints to send traffic to. In this example, a Deployment creates Pods whose labels match the Service's selector. Services can also select Pods managed by other controllers, and a Service without a selector can use explicitly managed EndpointSlices.

A Deployment manages ReplicaSets to create and update its Pods.

Creating a Deployment

A user can create a deployment directly using the Kubernetes API, or by using a configuration file like the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tensorboard
spec:
  selector:
    matchLabels:
      app: tensorboard
  template:
    metadata:
      labels:
        app: tensorboard
    spec:
      containers:
        - name: tensorboard
          image: tensorflow/tensorflow:2.19.0
          command: ["tensorboard"]
          args: ["--logdir=/logs", "--host=0.0.0.0", "--port=6006"]
          ports:
            - containerPort: 6006
          readinessProbe:
            tcpSocket:
              port: 6006
          volumeMounts:
            - name: logs
              mountPath: /logs
      volumes:
        - name: logs
          emptyDir: {}

This example uses a versioned TensorFlow image and starts TensorBoard explicitly. The emptyDir is initially empty, so the UI should show that no dashboards are active until event files are available. For real training results, replace it with a volume containing TensorBoard event files and mount that data read-only where possible. emptyDir data is lost when its Pod is removed.

Save the manifest as deployment.yaml, create it, and wait for its Pod to become ready:

kubectl apply --namespace workloads-demo -f deployment.yaml
kubectl rollout status --namespace workloads-demo deployment/tensorboard --timeout=180s
kubectl get endpointslices --namespace workloads-demo -l kubernetes.io/service-name=tensorboard

The EndpointSlice should include the ready TensorBoard Pod's address. If it does not, compare the Service selector with the Pod labels and inspect the Pod's readiness and logs.

Exposing a Service

Depending on the service type, a service can be exposed in different ways. In the case of Service type ClusterIP, the service is only accessible from within the cluster, but can be port-forwarded to be accessed locally.

To port-forward the service tensorboard to the local machine, run the following command:

kubectl port-forward --namespace workloads-demo service/tensorboard 6006:6006

Open http://localhost:6006 while this command is running. The tunnel gives your machine access without changing the Service to a public endpoint.

Notes

Users interacting with Polyaxon's API will never need to create services directly, it's also important to note that services created manually will not be controlled and managed by Polyaxon.

Services managed by Polyaxon have a much simpler configuration, and they are created automatically when a user creates a notebook or a tensorboard. They are also exposed via Polyaxon's gateway, which means that users can access them via the Polyaxon's UI or CLI and do not have to learn about the underlying Kubernetes concepts. Users who are using one of our commercial offerings will also benefit from a complete authentication and authorization layer for all the services exposed.

Closing thoughts

When finished with the standalone examples, stop the port-forward and remove their disposable namespace:

kubectl delete namespace workloads-demo

Although Polyaxon provides a simple interface and configuration for scheduling and running workloads, it does not prevent users from interacting with the underlying Kubernetes API. Our specification provides full access to customize the underlying Kubernetes resources via:

• environment
• termination
• runtime