OpenTelemetry Collector for ML platforms

The OpenTelemetry Collector is a vendor-neutral service for receiving, processing, and exporting telemetry. It gives platform teams a controlled boundary between instrumented applications and one or more observability backends.

That boundary is valuable for ML systems, where telemetry spans ordinary services, Kubernetes infrastructure, accelerators, batch workloads, and model or agent behavior. The Collector moves and transforms signals; it does not create a useful observability strategy by itself.

Understand the pipeline

The official OpenTelemetry Collector documentation describes a pipeline from receivers through processors to exporters:

• Receivers ingest telemetry using protocols or integrations such as OTLP.
• Processors batch, filter, sample, enrich, redact, or protect the pipeline.
• Exporters send data to a backend or another Collector.
• Connectors join pipelines by acting as an exporter on one side and a receiver on another.
• Extensions provide operational capabilities such as health endpoints or authentication.

Components do nothing until they are enabled in a service pipeline. Keep metrics, logs, and traces explicit because their supported components and failure behavior can differ.

Choose agent, gateway, or both

An agent Collector runs close to the workload, often as a DaemonSet or sidecar. It can collect node-local signals, enrich records with local metadata, and reduce traffic before export.

A gateway Collector runs as a shared service. It centralizes policy, credentials, tail sampling, routing, and backend export. It also becomes a scaling and availability boundary.

Many platforms use both: agents handle local collection and a gateway handles centralized processing. Avoid sidecars by default when a node agent can serve many Pods; every sidecar consumes resources and shares the workload lifecycle.

Size replicas, load balancing, queues, and disruption policy for the selected pattern. Stateful processing such as tail sampling requires additional care because related spans must reach the right Collector instance.

Protect memory and backpressure

Telemetry spikes often coincide with incidents, exactly when losing evidence is most harmful. Configure memory protection, batching, sending queues, retry behavior, and persistent queueing where the durability requirement justifies it.

Understand the overload sequence. If a backend slows, exporter queues grow; memory rises; receivers may apply backpressure or data may be dropped. Monitor accepted, refused, sent, retried, and dropped telemetry by signal and exporter.

Do not retry permanently invalid data forever. Separate transient backend failure from schema, authentication, or configuration errors that need intervention.

Control enrichment and cardinality

Add resource attributes that support stable correlation: service, environment, cluster, namespace, workload class, and version. Normalize naming at the Collector boundary when instrumentation libraries differ.

Be cautious with Kubernetes metadata enrichment. Pod UIDs, run IDs, user labels, prompts, file paths, and experiment parameters can create high cardinality or expose sensitive content. Use metrics for bounded aggregation and keep precise identifiers in traces, logs, or Polyaxon run metadata.

Attribute processors can delete or transform sensitive fields, but redaction should happen as early as practical. Once a credential or personal value reaches several exporters, removing every copy becomes much harder.

Route signals by purpose

One pipeline does not need to send every record to every backend. Platform metrics may go to an operational time-series system, security-relevant logs to protected retention, traces to a tracing backend, and selected model or agent evaluations to a specialized workflow.

Document routing and failure expectations. Ask whether one exporter failure should block another, whether development data can enter production storage, and which tenants can query each destination.

Use OTLP between application instrumentation and the collection tier when possible. It reduces application coupling to a specific analytics vendor and lets platform policy change without rebuilding every workload.

Observe the Collector

Track Collector health as part of the platform:

• CPU and memory use;
• receiver acceptance and refusal;
• processor drops and sampling decisions;
• exporter queue size, failures, retries, and latency;
• configuration reload or rollout state;
• network and backend availability.

Run a synthetic telemetry path through the complete pipeline. A healthy process with a broken route or invalid credential can otherwise create silent blind spots.

Version configuration with the Collector distribution and components in use. Contrib components may have different maturity and stability levels; review release notes before upgrades.

Connect infrastructure to ML context

OpenTelemetry can connect request paths across APIs, model gateways, retrieval services, and supporting infrastructure. Polyaxon connects those signals to the lifecycle of ML operations: projects, code, data, parameters, metrics, logs, lineage, and artifacts.

Use Polyaxon platform observability for the services managing workloads and run logging for operation output. Our AI observability guide covers the additional evaluation and behavior signals needed for generative systems.

The Collector should remain a dependable telemetry control plane: accept known protocols, enforce bounded transformations, survive downstream pressure, and route evidence safely. The meaning of that evidence still comes from the objectives and ML context around it.