OpenTelemetry for Java ML services

Java services often sit on important ML paths: platform APIs, feature services, model gateways, streaming pipelines, and enterprise integrations. OpenTelemetry can provide consistent traces, metrics, and logs across these services without coupling the application to one observability backend.

The fastest starting point is usually automatic instrumentation. The best production result combines that coverage with deliberate resource identity, custom domain spans, controlled attributes, and an operated collection pipeline.

Choose the Java instrumentation path

The official OpenTelemetry Java documentation describes several approaches:

The Java agent is a strong baseline for existing services. The SDK offers more control but creates more application-owned configuration. Libraries should normally use the API rather than initialize a global SDK or select an exporter for their consumers.

Choose one primary initialization path. Running overlapping agent, starter, and manual configuration can create duplicate spans or competing providers.

Establish service identity first

Configure a stable service name, namespace, version, and deployment environment. Add cluster and Kubernetes workload metadata through a controlled enrichment layer.

Do not use Pod names or generated IDs as the service name. Those values fragment one logical service into many identities and make dashboards, sampling, and alerts difficult to maintain.

Keep resource attributes consistent across Java and non-Java services. A distributed trace becomes harder to query when each runtime uses a different name for the same environment or cluster.

Export through OTLP

Send telemetry over OTLP to an OpenTelemetry Collector rather than embedding destination-specific credentials and exporters in every application. The Collector can batch, filter, redact, sample, route, retry, and protect downstream backends.

Treat export as a failure-prone dependency. Telemetry backpressure should not exhaust application memory or prevent the service from serving requests. Configure bounded queues and timeouts, and monitor dropped or failed exports.

Our OpenTelemetry Collector guide covers agent, gateway, memory, queue, and routing decisions.

Add spans for ML domain work

Automatic instrumentation can show HTTP, database, messaging, and framework calls. It cannot infer that a method loads a model, runs retrieval, validates an output, or publishes an artifact.

Add manual spans around meaningful work:

• resolving a model or run version;
• loading model weights or a feature snapshot;
• preprocessing and postprocessing;
• calling an inference backend;
• retrieval and reranking;
• policy and safety evaluation;
• persisting an output or audit record.

Use stable operation names and record outcomes as bounded attributes or span events. Avoid attaching prompts, full responses, feature vectors, SQL parameters, credentials, or arbitrary user content.

Propagate context through async Java

Thread pools, futures, reactive frameworks, messaging clients, and scheduled work can break trace context when callbacks execute elsewhere. Verify propagation with the actual concurrency model rather than assuming the agent covers every library and version.

For queued or batched work, consider span links instead of forcing a direct parent-child relationship across long delays. Preserve a durable Polyaxon run identifier separately so traces can connect to the longer ML lifecycle.

Test retries. Each attempt should be visible without turning one logical request into unrelated traces or hiding the cost of failed attempts.

Control overhead and JVM signals

Measure CPU, allocation, memory, startup time, latency, and export traffic under representative load. Instrumentation can add work in hot request paths, and excessive attributes or spans can increase garbage collection pressure.

JVM metrics such as heap, garbage collection, threads, class loading, and CPU provide valuable process context. Keep them separate from application outcomes. A healthy heap does not mean the model response is correct, and a slow request may originate in a remote dependency rather than the JVM.

Set sampling and batch policy according to traffic volume and incident needs. Validate that errors, slow paths, and selected workload classes remain observable.

Roll out safely on Kubernetes

Pin the agent or starter version and test it against the JDK, framework, libraries, and container image in use. Canary the change, compare telemetry shape and application performance, and support a fast disable path.

If the platform injects agents through an admission webhook, scope it narrowly. Webhook unavailability or an incompatible mutation can block every selected Pod from starting.

Keep configuration and secrets outside the application image. Use least-privilege credentials for the Collector endpoint and restrict access to trace data.

Connect Java telemetry to Polyaxon

Polyaxon platform observability connects service health to the workloads being submitted and tracked. Polyaxon run metadata preserves the code, inputs, parameters, metrics, logs, and artifacts behind a Java service's request.

Start with broad automatic coverage, then add manual spans where ML meaning begins. The result should let an operator move from a service-level symptom through Java execution and downstream calls to the exact model or run context that explains the outcome.