Polyaxon v3 is coming →

Using kubectl delete

Use kubectl delete safely for Kubernetes resources, files, labels, namespaces, force deletion, and cleanup workflows.

April 30, 2024by Polyaxon
Apr 30, 2024

Using kubectl delete

Use kubectl delete safely for Kubernetes resources, files, labels, namespaces, force deletion, and cleanup workflows.

Picture

kubectl delete is simple enough to be dangerous. It removes Kubernetes resources, and depending on the resource, controllers may recreate them, data may persist, or state may disappear.

Use it deliberately. In shared ML clusters, deleting the wrong object can interrupt someone else's run, remove a service endpoint, or hide the evidence you needed for debugging.

What is kubectl delete?

kubectl delete offers you a way to gracefully shut down and terminate Kubernetes resources by their filenames or specific resource names. The graceful termination of resources using the kubectl delete command is to be done with utmost discretion, because once the command has been run, there's no way to undo the deletion of the resource.

Kubernetes doesn't provide users with the ability to simply pause and resume resources. There are several ways to get around this limitation, one of which is to simply stop (delete) the desired resource to pause, and reapply the resource definition to the cluster to resume.

The kubectl delete command is especially handy during clean up after the deployment of an application. During development, the creation and deployment of resources is a top concern, but the management of these resources sometimes falls to the wayside after deployment.

Some of these resource-management responsibilities involve deleting unused pods. These pods, while running and unused in the Kubernetes cluster, take up some of the node's resources, thereby limiting other, more important, running pods.

In some cases, a Kubernetes node needs to be shut down, and the Kubernetes deployments and services will need to be terminated. Another resource-management concern could be the deletion of StatefulSets that manage pods that rely on durable storage configurations, such as persistent volumes.

Using kubectl delete

Before you begin with this tutorial, you need to have an understanding of the following Kubernetes concepts:

  • Pods
  • StatefulSets
  • ReplicaSets
  • Deployments
  • Services
  • kubectl command-line tool

To get started with using kubectl delete, you need to have a single-node Kubernetes cluster running with at least 4 GB of RAM available. You can create one locally with lightweight Kubernetes distributions, such as minikube, K3s, or MicroK8s, or you can provision one with a cloud provider such as Google Kubernetes Engine (GKE), Azure Kubernetes Service, or Amazon Elastic Kubernetes Service (EKS).

Kubectl delete basic use case

With your cluster set up, you can now begin with a basic use case of the kubectl delete command.

Deploy a sample NGINX deployment by creating a file named deployment.yaml, then adding the following code snippet:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 80

You then run the configurations on your Kubernetes cluster with the kubectl apply command:

kubectl apply -f ./deployment.yaml

The command above will create an NGINX deployment with a ReplicaSet and three NGINX pods deployed and running in the default namespace in your Kubernetes cluster. To verify this, you can simply perform the kubectl get pod command, and you'll get the following response:

kubectl get pod

# Output
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-7f4f6d9f5b-4z5zv   1/1     Running   0          2m
nginx-deployment-7f4f6d9f5b-6z5zv   1/1     Running   0          2m
nginx-deployment-7f4f6d9f5b-7z5zv   1/1     Running   0          2m

Now that you've successfully deployed NGINX to your Kubernetes cluster, the next step is to clean up after the deployment.

If you perform the command kubectl delete pods --all to delete all the NGINX pods in the default namespace, another set of pods will spring back up to replace the ones that were deleted. In order to prevent this from happening, you need to delete the deployment configuration using either the filename, deployment.yaml, or the deployment name, nginx-deployment, as configured in the metadata section in the YAML code snippet above.

The NGINX Deployment can be deleted in two ways:

# Delete Deployment using filename
kubectl delete -f ./deployment.yaml

  # Delete Deployment using metadata name:
kubectl delete deployment nginx-deployment

Once the deployment resource has been deleted, you shouldn't be able to see any pods, deployments, or ReplicaSets in the default namespace. You'll see only a single Kubernetes service active when you perform the kubectl get services command:

kubectl get services

# Output
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP

Please note that depending on the type of Kubernetes distribution you are running, the output shown above may differ slightly.

Kubectl delete advanced use case

In the previous section, you created a simple NGINX deployment, then deleted it properly. In this section, you'll look at a more advanced use case for the kubectl delete command, and see the steps involved in applying the command.

In this example, you'll deploy the popular WordPress CMS. To do this, you'll create the WordPress service, the wp-pv-claim PersistentVolumeClaim, and deploy the Wordpress app using the latest Docker Hub image.

Using your favorite text editor, create a file named wordpress.yaml and paste in the following YAML code:

apiVersion: v1
kind: Service
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  ports:
    - port: 80
  selector:
    app: wordpress
    tier: frontend
  type: LoadBalancer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wp-pv-claim
  labels:
    app: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: frontend
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: frontend
    spec:
      containers:
        - image: wordpress:5.9.3-apache
          name: wordpress
          env:
            - name: WORDPRESS_DB_HOST
              value: wordpress-db
            - name: WORDPRESS_DB_PASSWORD
              value: password
          ports:
            - containerPort: 8080
              name: wordpress
          volumeMounts:
            - name: wordpress-persistent-storage
              mountPath: /var/www/html
      volumes:
        - name: wordpress-persistent-storage
          persistentVolumeClaim:
            claimName: wp-pv-claim

Apply the above configuration to your cluster by running the kubectl apply command:

kubectl apply -f ./wordpress.yaml

# Output

service/wordpress created
persistentvolumeclaim/wp-pv-claim created
deployment.apps/wordpress created

Verify that the changes were executed correctly using the kubectl get all command:

kubectl get all

# Output

NAME                           READY   STATUS    RESTARTS   AGE
pod/wordpress-7f4f6d9f5b-4z5zv   1/1     Running   0          2m
pod/wordpress-7f4f6d9f5b-6z5zv   1/1     Running   0          2m
pod/wordpress-7f4f6d9f5b-7z5zv   1/1     Running   0          2m

NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/wordpress    LoadBalancer

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/wordpress   3/3     3            3           2m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/wordpress-7f4f6d9f5b   3         3         3       2m

After verifying that the WordPress deployment was successful, you can proceed to cleanup.

As before, trying to delete the pods is futile, since the deployment specification will recreate them as many times as necessary. Fortunately, you only have to use one command to remove the newly created deployment, service, and PVC:

kubectl delete -f ./wordpress.yaml
service "wordpress" deleted
persistentvolumeclaim "wp-pv-claim" deleted
deployment.apps "wordpress" deleted

After running the command above, you should see that the WordPress deployment, service, and PVC have been deleted. You can verify this by running the kubectl get all command again:

kubectl get all

# Output
No resources found in default namespace.

As you can see, the kubectl delete command is suitable for both simple and more advanced use cases.

However, note that this was possible because a single YAML file was used to configure all the moving parts. This may or may not be a good idea, as you could mistakenly delete even data stored on a persistent volume. This leads us to the next section, best practices.

Best practices

A single misused kubectl delete command can destroy a whole application and render it useless. There are no safety measures, such as prompts to ask if you truly want to delete the resource or not, attached to the command. Before you perform an operation with the kubectl delete command, you need to know exactly what you want to delete and how you want it deleted.

You can use the kubectl delete --help command to view a list of flags and options that can be applied.

Know when and how to use kubectl delete

Understanding when to delete Kubernetes resources requires that you know your application's tolerance for the unavailability of one or more of the resources. If one of the resources is taken down, your application should still be able to function.

Similarly, knowing how to delete resources is essential to keeping your services and applications running. For instance, deleting resources with the --force flag set to true should be avoided, as this may result in resource inconsistency, leading to application runtime failure or data loss. Unless you're very certain that doing this won't create problems for your application, this should be used only as a last resort.

Don't use kubectl delete unless absolutely needed

It is strongly advised that you use the kubectl delete command sparingly, unless during important operations such as clean-up operations for resource optimization once all deployments are done with.

It's also advisable to hold off on deleting PersistentVolumeClaims until all necessary data is retrieved or copied, using the kubectl cp command from the mountPath on to your host machine.

Final thoughts

kubectl delete is not a cleanup shortcut to use lazily. Know whether a controller will recreate the object, whether state is attached, and whether the resource is still needed for debugging.

In Polyaxon-managed environments, prefer platform-level controls for runs and services when available. Drop to kubectl when you need cluster-level intervention and know exactly what object you are removing.