OpenTelemetry Collector for ML platforms

The OpenTelemetry Collector is a vendor-neutral service for receiving, processing, and exporting telemetry. It gives platform teams a controlled boundary between instrumented applications and one or more observability backends.

That boundary is valuable for ML systems, where telemetry spans ordinary services, Kubernetes infrastructure, accelerators, batch workloads, and model or agent behavior. The Collector moves and transforms signals; it does not create a useful observability strategy by itself.

Understand the pipeline

The official OpenTelemetry Collector documentation describes a pipeline from receivers through processors to exporters:

Receivers ingest telemetry using protocols or integrations such as OTLP.
Processors batch, filter, sample, enrich, redact, or protect the pipeline.
Exporters send data to a backend or another Collector.
Connectors join pipelines by acting as an exporter on one side and a receiver on another.
Extensions provide operational capabilities such as health endpoints or authentication.

Pipeline components do nothing until they are enabled in a service pipeline; extensions are enabled separately under service.extensions. Keep metrics, logs, and traces explicit because their supported components and failure behavior can differ.

Follow one synthetic trace

Start with a path you can inspect: a local sender → OTLP/HTTP receiver → memory protection → attribute removal → batching → debug exporter. This example requires Docker and Python 3. Clone the examples repository and enter the Collector directory:

git clone https://github.com/polyaxon/polyaxon-examples.git
cd polyaxon-examples/blog/otel-collector

If you already have the repository, use this directory in your existing checkout. It contains collector.yaml and send-trace.py. Run the commands below from that directory so the configuration mount resolves correctly. The complete Collector configuration is:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 128
    spike_limit_mib: 32
  attributes/remove-demo-email:
    actions:
      - key: user.email
        action: delete
  batch:
    timeout: 1s
    send_batch_size: 128

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, attributes/remove-demo-email, batch]
      exporters: [debug]

The receiver binds inside the container, while Docker publishes it only on the host's loopback address. The example uses the Contrib distribution for the attributes processor and pins the image to 0.160.0; keep the image and configuration versioned together. The Collector Docker guide documents image selection and configuration mounting.

docker run --rm --name ml-otel-demo \
  --memory=256m \
  --publish 127.0.0.1:4318:4318 \
  --volume "$PWD/collector.yaml:/etc/otelcol-contrib/config.yaml:ro" \
  otel/opentelemetry-collector-contrib:0.160.0 \
  --config=/etc/otelcol-contrib/config.yaml

After the Collector starts, use a second terminal in the example directory:

python3 send-trace.py

The sender creates one synthetic span with a fresh timestamp and trace ID using the OTLP/HTTP JSON format. Its one-millisecond duration describes a fabricated span; it is not an inference benchmark. Inspect the Collector terminal after the batch flushes:

Check	Expected evidence
Receiver accepted the request	The sender prints a trace ID without an HTTP or partial-success error
The trace reached the exporter	That trace ID and `synthetic-model-request` appear in debug output
Context survived processing	`service.name=ml-collector-demo` and `ml.operation.id=example-operation` are present
The specified attribute was removed	The span's `user.email` attribute is absent

The application supplied the operation attribute; it is not inferred by the Collector or automatically populated by Polyaxon. In an instrumented workload, supply the actual run identifier from your application's run context and retain it in traces or logs for navigation.

This processor removes one known span attribute. It does not scrub arbitrary span names, resource attributes, events, or log bodies. The attributes processor reference explains its scope. Debug export prints telemetry, so use synthetic data for this walkthrough and replace it with your backend's exporter for operational traffic. Stop the Collector with Ctrl+C when finished.

Choose agent, gateway, or both

An agent Collector runs close to the workload, often as a DaemonSet or sidecar. It can collect node-local signals, enrich records with local metadata, and reduce traffic before export.

A gateway Collector runs as a shared service. It centralizes policy, credentials, tail sampling, routing, and backend export. It also becomes a scaling and availability boundary.

Many platforms use both: agents handle local collection and a gateway handles centralized processing. Avoid sidecars by default when a node agent can serve many Pods; every sidecar consumes resources and shares the workload lifecycle.

Size replicas, load balancing, queues, and disruption policy for the selected pattern. Stateful processing such as tail sampling requires additional care because related spans must reach the right Collector instance.

Protect memory and backpressure

Telemetry spikes often coincide with incidents, exactly when losing evidence is most harmful. Configure memory protection, batching, sending queues, retry behavior, and persistent queueing where the durability requirement justifies it.

Understand the overload sequence. If a backend slows, exporter queues grow; memory rises; receivers may apply backpressure or data may be dropped. Monitor accepted, refused, sent, retried, and dropped telemetry by signal and exporter.

Do not retry permanently invalid data forever. Separate transient backend failure from schema, authentication, or configuration errors that need intervention.

Control enrichment and cardinality

Add resource attributes that support stable correlation: service, environment, cluster, namespace, workload class, and version. Normalize naming at the Collector boundary when instrumentation libraries differ.

Be cautious with Kubernetes metadata enrichment. Pod UIDs, run IDs, user labels, prompts, file paths, and experiment parameters can create high cardinality or expose sensitive content. Use metrics for bounded aggregation and keep precise identifiers in traces, logs, or Polyaxon run metadata.

Attribute processors can delete or transform sensitive fields, but redaction should happen as early as practical. Once a credential or personal value reaches several exporters, removing every copy becomes much harder.

Route signals by purpose

One pipeline does not need to send every record to every backend. Platform metrics may go to an operational time-series system, security-relevant logs to protected retention, traces to a tracing backend, and selected model or agent evaluations to a specialized workflow.

Document routing and failure expectations. Ask whether one exporter failure should block another, whether development data can enter production storage, and which tenants can query each destination.

Use OTLP between application instrumentation and the collection tier when possible. It reduces application coupling to a specific analytics vendor and lets platform policy change without rebuilding every workload.

When replacing the example's debug exporter, configure the backend's OTLP endpoint, TLS trust, and authentication, then enable that exporter in service.pipelines.traces.exporters. Inject credentials from your deployment's secret mechanism. Repeat the synthetic request and find its exact trace ID in the destination: receiver acceptance alone cannot prove that the backend received or retained it. Size sending queues and retry limits against the outage duration you intend to tolerate.

Observe the Collector

Track Collector health as part of the platform:

CPU and memory use;
receiver acceptance and refusal;
processor drops and sampling decisions;
exporter queue size, failures, retries, and latency;
configuration reload or rollout state;
network and backend availability.

Run a synthetic telemetry path through the complete pipeline. A healthy process with a broken route or invalid credential can otherwise create silent blind spots.

Version configuration with the Collector distribution and components in use. Contrib components may have different maturity and stability levels; review release notes before upgrades.

Connect infrastructure to ML context

OpenTelemetry can connect request paths across APIs, model gateways, retrieval services, and supporting infrastructure. Polyaxon connects those signals to the lifecycle of ML operations: projects, code, data, parameters, metrics, logs, lineage, and artifacts.

Use Polyaxon platform observability for the services managing workloads and run logging for operation output. Our AI observability guide covers the additional evaluation and behavior signals needed for generative systems.

Continue with distributed tracing for ML platforms for application instrumentation and context propagation. The Collector configuration above handles transport and processing after the application creates those spans.

The Collector should remain a dependable telemetry control plane: accept known protocols, enforce bounded transformations, survive downstream pressure, and route evidence safely. The meaning of that evidence still comes from the objectives and ML context around it.