Polyaxon v3 is coming →

Prometheus exporters: tutorial and best practices

Learn how Prometheus exporters expose third-party metrics, how to build a simple exporter, and what practices keep metrics useful.

November 12, 2024by Polyaxon
Nov 12, 2024

Prometheus exporters: tutorial and best practices

Learn how Prometheus exporters expose third-party metrics, how to build a simple exporter, and what practices keep metrics useful.

Picture

Prometheus works best when targets expose metrics directly. Many systems do not. Exporters fill the gap by translating application, database, or infrastructure signals into Prometheus-friendly metrics.

The exporter itself should stay boring. Collect useful metrics, label them clearly, expose them reliably, and avoid turning the monitoring path into another fragile service.

What is a Prometheus exporter?

A Prometheus exporter aggregates and imports data from a non-Prometheus to a Prometheus system. An exporter is considered essential in any cloud-native ecosystem that includes applications that don't expose Prometheus-supported metrics by default. A Prometheus exporter acts as a proxy between such an applications and the Prometheus server. Exporters use a simple, text-based, key-value format to scrape and expose metrics over time, via HTTP, for aggregation and decision-making.

How do Prometheus exporters work?

Prometheus implements the HTTP pull model to gather metrics from client components. For event-based monitoring, the Prometheus client relies on an exporter that acts as an abstraction layer between the application and the Prometheus server.

A Prometheus exporter's working mechanism typically involves the following:

  • Providing a target endpoint that the Prometheus server periodically queries for metrics.
  • Extracting metrics data from a non-Prometheus application.
  • Transforming captured data into a Prometheus ingestible format by leveraging client libraries.
  • Initiating a web server to display metrics through a URL.

Prometheus exporter implementation types

In a complex ecosystem of multiple stateful and stateless applications, there are two approaches to implementing Prometheus exporters for comprehensive metrics collection. These include:

Application built-in exporters

Built-in exporters are used when the client system natively exposes key metrics, including request rates, errors, and duration. Common use cases include:

  • Used to develop an application from scratch: developers assess and integrate Prometheus client requirements from the initial phases of application design.
  • Integrated into existing applications: teams modify code to add specific Prometheus client capabilities.

Third-party/standalone exporters

Standalone exporters are used to expose metrics so they can be collected and processed externally. In such instances, applications typically expose metrics through a metric API or logs. Standalone exporters are also provisioned in setups where the exposed data relies on transformation and aggregation by an external service.

How to set up a Prometheus exporter for monitoring and alerts

Although there are multiple ways of building exporters, Prometheus ships with a Python library to support the development of metric collectors and exporters in a few simple steps. In this section, we'll discuss how to build a basic exporter for a Kubernetes cluster using a Python script. The demo workflow essentially includes the steps for building the exporter using Python, building a Docker image, and using the image in a cluster deployment object for exposing metrics.

Prerequisites

An existing Kubernetes cluster. Docker service installed with the CLI logged into Docker Hub...

Creating the exporter using Python script

First, create the working directory using a command in this format:

$ mkdir custom-exporter

Navigate to the directory:

Create the directory for Python scripts and navigate to it:

$ cd code
$ vim collector.py

The exporter script is divided into three parts. The first part imports all the dependencies required to implement a metrics exporter. To perform this step, add the following code to collector.py:

import time
from prometheus_client.core import GaugeMetricFamily, REGISTRY, CounterMetricFamily
from prometheus_client import start_http_server

The second part defines the custom exporter class to be built and implements the exporter object for each desired metric. Add the following code to collector.py to build the custom exporter class:

class CustomCollector(object):
    def __init__(self):
        pass

To define objects for metric exporters, first add the following code to declare the exporter objects, Once that's done, you add two objects for the metric exporters. To add a gauge metric for memory usage, add the following code to collector.py: The second object is a counter metric that enumerates the number of HTTP requests made to the cluster. To add the counter metric object, add the following code to collector.py:

def collect(self):
    g = GaugeMetricFamily("custom_metric", 'Description of custom metric', labels=['label'])
    g.add_metric(["label_value"], 10)
    yield g

Once the metric objects are added to the script, the third and final section of the script defines the initiation of the Prometheus server and provides an endpoint for scraping metrics.


if __name__ == '__main__':
    start_http_server(8000)
    REGISTRY.register(CustomCollector())
    while True:
        time.sleep(1)

To build exporters when being deployed, this code will rely on Prometheus client libraries and modules. To store this data within the image, create a pip-requirements.txt file that references Prometheus client libraries:

prometheus_client

Building the Docker image

Navigate back to the Project directory:

$ cd ..

Create the Dockerfile to be used in building container images:

$ vim Dockerfile

Add details of the working directory and environment to the Dockerfile, as shown below:

FROM python:3.6

ADD code /code
RUN pip install -r /code/pip-requirements.txt

WORKDIR /code
ENV PYTHONPATH '/code/'

CMD ["python" , "/code/collector.py"]

Once the Dockerfile is saved, run the following command to build the container image:

$ docker build -t <docker-hub-username>/custom-exporter .

Confirm the creation of the image by running the command:

$ docker images

Which should return a response similar to:

REPOSITORY	TAG	IMAGE ID	CREATED	SIZE
<docker-hub-user>/custom-explorer	latest	4de8857db072	38 seconds ago	908MB
python	3.6	54260638d07c	8 months ago	902MB

Push the image to Docker Hub by running the command:

$ docker push <docker-hub-username>/custom-exporter

Deploying the exporter in a Kubernetes cluster

Once the Docker image is successfully deployed, the next step is to build a service and deployment object that uses the Docker image to create a working operator.

Create and navigate to the folder to host the templates:

$ mkdir templates
$ cd templates

Create the deployment manifest:

$ vim custom-exporter-deployment.yaml

Add the following details to the deployment manifest:


apiVersion: v1
kind: Service
metadata:
  name: darwin-custom-collector-service
spec:
  selector:
    app: darwin-custom-collector
  type: NodePort
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8000

Apply both the configurations:

$ kubectl apply -f custom-exporter-deployment.yaml

Confirm the deployment by running the command:

$ kubectl get pods

NAME	READY	STATUS	RESTARTS	AGE
darwin-custom-collector-75cff7d89f-r8nml	1/1	Running	0	32s

To access Prometheus from the localhost over port 8080, run a command similar to the one below (make sure to replace the pod name shown below with the one in your cluster):


$ kubectl port-forward darwin-custom-collector-75cff7d89f-r8nml 8080:9090

Once the exporter is up, you'll be able to access http://<your-machine-ip>:9113/metrics.

Best practices when using Prometheus exporters

Some best practices to adopt when using Prometheus exporters include:

Use an existing Prometheus exporter

Developers should use default Prometheus exporters built for collecting third-party metrics from different application types. Building a custom exporter not only adds to effort overhead but also introduces regressions into an application's core functions. To find the right Prometheus exporter, developers should evaluate the maturity of the exporter as an open-source project and the type of metrics it can expose. The Prometheus website provides a comprehensive exporters and integrations list that can help you identify the best choice for a deployment. Other third-party sites, such as PromCat.io, also provide curated exporter lists that are useful in choosing the right exporter for a complex cluster of different applications.

Use labels and annotations to help understand metrics

Each exporter collects and exposes a unique set of metrics. While the metric name is always verbose, comprehending the information it presents is often difficult at first glance. Some exporters use the OpenMetrics format to provide additional context about a metric out-of-the-box. In the absence of such information, attaching metadata through labels and annotations is recommended to offer context and meaning. Attaching metadata for Kubernetes object can be done with:

  • Labels are key-value pairs that help with grouping and querying objects. These can further be classified into:
    • Instrumentation labels for analyzing the information exported from inside an application.
    • Target labels for analyzing metrics aggregated from disparate sources in a full-scale deployment.
  • Annotations store information about objects that can be further used by internal tools and libraries.

Configure actionable alerts

Apart from capturing the right set of metrics, monitoring teams should set up alerts for events such as a metric drifting away from desired values. On a distributed cluster of numerous applications and continuous events, defining an alert strategy is a complex undertaking. If the metrics threshold is too low, the monitoring team gets overburdened with unnecessary alarms. But if the threshold is too high, the monitoring team may miss critical events, leading to system failure or an undesirable user experience. As a recommended practice, monitoring teams should configure notifications to ensure the perfect balance between an optimum user experience and timely responses to deployment issues.

Adopt an appropriate scaling mechanism

As deployments grow, leveraging numerous exporters and metrics introduces storage and visibility bottlenecks. Avoiding such bottlenecks often relies on comprehensive observability that conforms to the gradual growth of a cluster environment. Considering Prometheus' innate inefficiency of horizontal scaling, administrators should adopt an appropriate scaling mechanism while proactively accounting for the rising number of services, metric cardinality, and memory usage. Although manual scaling using federation is one option, the approach typically introduces manual overhead in configuration and maintenance. As a recommended approach, organizations should adopt cloud-native, long-term storage (LTS) for automatic persistence of metric data.

Administer reliable metric access privileges

As exporters expose sensitive information about applications, services, and hosts, determining who has access to these metrics and how they can use it is critical. HTTP endpoints and logs of Prometheus are also susceptible to vector attacks since they expose a cluster's operating information and debugging patterns. Organizations should use role-based access controls and implement security filters to ensure that only authorized users can access metrics and execute reports. Another recommended practice is to make use of Prometheus' default authorization and TLS encryption features to prevent API endpoints from exposing sensitive metric data.

Final thoughts

Exporters are glue. Good glue is boring: stable endpoints, clear labels, limited cardinality, and alerts that point to action instead of panic.

Polyaxon teams can use Prometheus-style metrics alongside run metadata to connect infrastructure health with ML workload behavior. The exporter gives the metric; the platform context tells you why it matters.