Dec 17, 2024

Kubernetes operators: a complete guide

Learn how Kubernetes operators extend the API to manage application lifecycle, stateful systems, upgrades, backups, and automation.

Operators package operational knowledge into Kubernetes controllers. Instead of a human remembering every upgrade, backup, and reconciliation step, an operator watches custom resources and acts on them.

That is powerful, but it is not magic. Operators are code with permissions. Use them when they remove real operational burden, and inspect what they own before handing them production state.

What are Kubernetes operators?

Someone who's managing cluster operations has an in-depth understanding of a specific domain, including how to deploy a service, the cluster's normal behavior, and how to troubleshoot issues. The Kubernetes operator pattern was developed to automate intricate cluster operations that require iterative, manual efforts.

Kubernetes operators are built to automate scaling, upgrading, and configuration management for stateful applications. Stateful workloads such as monitoring systems and databases require domain-specific knowledge that isn't natively offered by Kubernetes. To help with this, operators encode the domain knowledge into application extensions, which makes the management of stateful applications standardized, scalable, and repeatable.

How operators manage applications in Kubernetes

Operators in Kubernetes are custom controllers that rely on custom resources (CRs) to manage services and cluster components. Similar to controllers, operators use control loops to compare a cluster state with the desired state, and then to implement a state change if there's a difference. High-level cluster configurations and directives are specified within the operator's custom resource definition (CRD), where the operator translates these directives into cluster actions based on Kubernetes automation principles embedded within the operator's logic.

Kubernetes operator use cases

Kubernetes operators eliminate tedious stateful application management tasks, simplifying the deployment and orchestration of critical cluster services. In a typical Kubernetes ecosystem, operators can be used for implementing both Day 1 (installation and configuration) and Day 2 ( maintenance, management and optimization) tasks. Uses of operators in Kubernetes clusters include:

Deploying apps on-demand

As operators encapsulate and package stateful applications for reuse, they can be deployed on demand across different stages of cluster operations. Operators also automate various post-provisioning tasks such as internal configuration, ingress and egress communication configuration, and capacity scaling.

Handling smooth code upgrades

Operators can be deployed to detect code changes in database schemas and application repositories, and to replicate the changes across containerized cluster workloads. Operators are also considered critical in a continuous delivery framework that uses declarative configuration upgrades to automate the implementation of changes to cluster configuration.

Enabling service discovery for unsupported Kubernetes APIs

Operator custom resources can be used to set up service-to-service communication by provisioning a pluggable policy layer. With operators, cluster administrators can configure APIs to implement access controls for smooth networking and service discovery of non-Kubernetes-native applications.

Performing backups and restoration

Cluster administrators can create CRs for volume snapshots to enable smooth backups and restoration of a cluster. This is typically achieved by configuring an operator to create a backup file and store it in a location configured within a Backup resource. A custom resource for restoration (Restore) can further be created with the backup file as a specification to automate service and data restoration during outages.

Hybrid environment management

When running Kubernetes clusters on multi-cloud instances, cluster administrators can use Kubernetes operators to simplify management and configuration tasks without acquiring platform expertise. In hybrid cloud clusters, operators are critical components for complex cluster management, offering agility, reliability, and consistency while adhering to each platform's best practices.

Using operators in a Kubernetes ecosystem

Operators in Kubernetes help with the efficient deployment and management of stateful applications, typically on behalf of a cluster administrator or site reliability engineer. An operator extends the Kubernetes API without requiring modifications to the Kubernetes code base. By bridging the gap between Kubernetes automation capabilities and external applications, operators ensure that all aspects of a software's lifecycle are integrated into the cluster and invoked as needed.

Operator deployment options

There are two primary approaches to deploying operators in a Kubernetes cluster:

Third-party operators

These are pre-built solutions that extend the Kubernetes API for managing third-party applications as native objects of a Kubernetes cluster. Third-party operators can be installed with standard, pre-configured settings or can be modified by cluster administrators to suit specific use-cases. OperatorHub (project by Red Hat) and Artifact Hub (project by CNCF) are two community-driven public repositories that allow operators to be discovered, downloaded, or uploaded along with their base documentation and information on vendor support.

Custom operators

Cluster management teams can also build their own operators for use cases that don't have a third-party operator available. Building custom operators is a complex undertaking, as it requires expert programming skills coupled with a deep knowledge of Kubernetes controllers and reconciliation loops. While these custom controllers can be implemented in any language that enables server-client communications, a few platforms are most commonly used:

• Go: The most popular language for building Kubernetes operators, Go is used for its simplicity, performance, and compatibility with Kubernetes libraries.
• Ansible: Ansible is a configuration management tool that can be used to build operators with minimal coding. Ansible operators are written in YAML and Python, and are suitable for operators that require a high level of customization.
• Helm: Helm is a package manager for Kubernetes that can be used to create operators with Helm charts. Helm operators are written in YAML and Go, and are suitable for operators that require a high level of customization.
• Operator SDK: The Operator SDK is a framework that helps developers build operators in Go, Ansible, or Helm. It provides tools for scaffolding, building, and testing operators, and is suitable for operators that require a high level of customization.
• KUDO: The Kubernetes Universal Declarative Operator (KUDO) is a toolkit for building operators with a focus on declarative configuration. KUDO operators are written in YAML and Go, and are suitable for operators that require a high level of customization.
• Operator Lifecycle Manager (OLM): OLM is a tool that helps manage the lifecycle of operators in a Kubernetes cluster. It provides tools for installing, upgrading, and managing operators, and is suitable for operators that require a high level of customization.
• Operator Framework: The Operator Framework is a set of tools that help developers build, test, and package operators. It provides tools for scaffolding, building, and testing operators, and is suitable for operators that require a high level of customization.

6 popular Kubernetes operators

Prometheus

Prometheus is a widely adopted cloud-native monitoring framework. Similar to any other Kubernetes deployment object, the Prometheus operator enables the deployment and management of Prometheus monitoring services in a Kubernetes cluster. The operator comes preconfigured with some fundamental Prometheus components and services, such as:

• AlertManager
• Grafana
• Node exporter
• Versions
• Retention policies
• Metric data persistence
• Deployment replicas

The Prometheus operator also auto-generates Prometheus target configurations based on Kubernetes queries, eliminating the need to learn Prometheus Query Language (PromQL).

Grafana

Similar to the Prometheus operator, the Grafana operator simplifies the deployment, configuration, and management of Grafana instances in Kubernetes clusters. When compared with a standalone Grafana instance, the Grafana operator offers fine-grained control over managing Grafana service through Kubernetes resources such as ConfigMaps and CRDs.

The operator also supports advanced features for Grafana instances, including:

• Data source recovery
• Dashboard creation and configuration through YAML
• Ingress automation
• Provisioning notification channel

RBAC manager

Kubernetes relies on role-based access control (RBAC) to administer roles that are bound to containers for permission management. The RBAC manager operator uses the RBACDefinition custom resource to reduce operational overhead for managing role binding and service accounts. The operator uses declarative configuration of RBAC to eliminate errors in role management changes and simplifies scaling of complex clusters.

Elastic cloud on Kubernetes operator

The Elastic Kubernetes Operator is the official open-source operator for Elasticsearch, Logstash, and Kibana (also known as ELK or Elastic Stack). The Elastic Kubernetes Operator helps to automatically deploy and manage the Elastic Stack on Kubernetes clusters for the collection, formatting, search, visualization, and analysis of data from any source. The operator is preconfigured to offer a number of advanced features, such as:

• Scheduled backups
• Upgrades
• Monitoring
• Dynamic scaling of storage
• Scaling cluster capacity

AWS controllers for Kubernetes (ACK)

ACK is a powerful tool that automates the management of AWS services directly using the Kubernetes API. The tool packages controllers for each AWS service in a separate container image within a publicly available repository. Each image corresponds to a single controller, making it easy to invoke supporting services such as message queues and databases. With the ACK operator, cluster administrators can take advantage of AWS features to deploy and configure resources for Kubernetes workloads without having to initiate the services outside the cluster.

ISTIO

Istio is a popular service mesh platform that automates cluster network functions in a flexible, transparent, and language-independent way. The Istio operator manages the installation, management, and upgrading of the Istio service mesh in a Kubernetes cluster, eliminating the manual overhead of using a Helm or Istioctl installation. The operator relies on the IstioOperator resource to detect configuration changes and updates the Istio installation based on the desired state of profile values.

Deploying an operator in Kubernetes

There are multiple ways to deploy operators into Kubernetes clusters. While most third-party solution vendors provide command line tools that expedite the installation of operators, experienced teams can also deploy operators manually using Helm templates to package CRDs and CRs for the installation and configuration of operators.

Although manual deployment of operators through Helm charts may sound tricky, the steps for doing so are typically straightforward. Here, we outline the steps for installing a Prometheus operator using Helm charts:

Prerequisites

• An existing Kubernetes cluster with Helm installed
• Helm CLI installed on your local machine

As the first step, download Prometheus templates into your operating cluster's control plane by running the command:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

>> "prometheus-community" has been added to your repositories

To ensure that the repository is using the updated version, update the repository using the command:

$ helm repo update

>> ...Successfully got an update from the "prometheus-community" chart repository

Install Prometheus into the cluster using the command:

$ helm install prometheus prometheus-community/prometheus-operator

Final thoughts

Operators are useful when they encode real operational knowledge: upgrades, backups, failover, reconciliation, and state management. They are harmful when installed casually with broad permissions and no owner.

Polyaxon integrates with Kubernetes operators for specialized workloads where they make sense, but the same rule applies: use the abstraction to reduce operational risk, not to hide risk from view.