Use eBPF to improve Kubernetes monitoring

Kubernetes already exposes object state, resource metrics, logs, and events. Those signals explain what the control plane believes should happen and what applications choose to report. They do not always show what the Linux kernel actually did between a process, the network, and the filesystem.

eBPF can close that gap. It lets approved programs attach to kernel hooks and collect structured data about system calls, packets, sockets, scheduling, and other low-level activity. The result can be powerful Kubernetes observability without modifying every application image.

Start with the question, not the technology

An eBPF deployment should answer a question that existing telemetry cannot answer reliably. Useful examples include:

• Which Pod initiated a connection to an unexpected destination?
• Where is network latency introduced between two services?
• Which process is repeatedly opening a missing file?
• Are connection resets occurring in the application, the node, or the network path?
• Which workloads are generating excessive system calls or DNS traffic?

If CPU utilization, Pod state, or an application error counter already answers the question, use the simpler signal. eBPF is an additional observation layer, not a replacement for Prometheus metrics, structured logs, traces, or Kubernetes events.

Understand the execution model

An eBPF program is loaded into the kernel and attached to one or more hooks. Before it runs, the kernel verifier checks that the program follows safety constraints. Programs exchange data with user-space agents through maps and other mechanisms; the agent enriches, aggregates, and exports the results.

That architecture matters operationally. Collection is distributed across nodes, while interpretation usually happens in a daemon, collector, or backend. BPF Type Format provides type information that helps tools work across compatible kernel structures. Our guide to BTF for eBPF observability explains why that portability layer matters.

“Runs in the kernel” does not mean “zero overhead” or “risk free.” Hook frequency, captured payload, map size, aggregation, and export volume all affect cost.

Map kernel activity back to Kubernetes

Raw process IDs and IP addresses are short-lived in a cluster. A useful eBPF pipeline enriches observations with stable context such as cluster, namespace, workload owner, Pod UID, container, node, and image digest.

Use the Pod UID rather than only the Pod name when identity must survive deletion and recreation. Preserve the controller owner—Job, Deployment, StatefulSet, or operator resource—so a signal can be assigned to the configuration that created it.

For ML workloads, include bounded identifiers that connect infrastructure activity to the experiment or operation. A Polyaxon run ID is useful in logs or trace attributes, but avoid turning every unique run, path, or request into a Prometheus label. High-cardinality enrichment can make the monitoring system more expensive than the problem it observes.

Choose a deployment pattern deliberately

Node-level collectors commonly run as a DaemonSet because they need one instance on each eligible node. Limit the node pool and privileges to what the collector actually requires. GPU nodes, general-purpose nodes, and sensitive pools may need different policies.

Before rollout, review:

• supported kernel versions and required kernel configuration;
• BTF availability or the tool's fallback strategy;
• required capabilities, host mounts, and security context;
• compatibility with the container runtime and network plugin;
• memory, CPU, map, and telemetry-export limits;
• upgrade and rollback behavior during node-image changes.

Treat the agent as privileged infrastructure. Pin its image by digest, verify provenance, restrict who can change it, and audit its configuration.

Control overhead and data volume

Measure the collector itself before enabling broad capture. A probe attached to a very frequent function can create significant work even when each event is small. Capturing payloads or stack traces is more expensive than counting events in the kernel and exporting aggregates.

Prefer a staged approach:

1. Enable a narrow signal on a small node pool.
2. Record collector CPU, memory, dropped events, map pressure, and export rate.
3. Validate that the signal resolves a real incident or operational question.
4. Add sampling, filtering, and in-kernel aggregation.
5. Expand only after setting budgets and ownership.

Do not retain sensitive arguments, file contents, or packet payloads simply because the tool can capture them. Redact close to collection and document who can query the resulting data.

Combine eBPF with other signals

eBPF is strongest when it narrows a hypothesis built from higher-level telemetry. A useful investigation might move through these layers:

1. Polyaxon shows a training operation slowed after a new image revision.
2. Prometheus shows normal GPU activity but increased network retransmits on one node pool.
3. Kubernetes events show no scheduling or volume errors.
4. eBPF network data identifies connections stalling on a particular destination and path.
5. Application traces or logs confirm the affected data-loading request.

Each signal answers a different question. Kubernetes state describes orchestration, metrics describe trends, logs describe discrete application records, traces describe request paths, and eBPF describes selected kernel activity.

Use eBPF with Polyaxon

Polyaxon provides the workload identity and execution history needed to interpret node-level observations. Standardize labels and annotations that connect Pods to projects, components, operations, images, and owners, then enrich observability records without exposing secrets or unbounded parameters.

Use scheduling presets to keep observability agents and ML workloads on the intended node pools. Keep collector privileges in platform-owned configuration rather than granting experiments broad host access.

The goal is not to collect every kernel event. It is to create a short, reliable path from a failed or slow ML operation to the responsible process, resource, network flow, or policy. eBPF earns its place when it makes that path materially clearer while staying inside a measured security and performance budget.