Kubernetes networking for ML workloads

An ML workload rarely uses only the network inside its container. Before useful work begins, it may pull an image, resolve a package mirror, read a dataset, download model weights, and contact a tracking service. During execution, distributed workers exchange data. At the end, the workload uploads artifacts and metrics.

Kubernetes networking provides the path for each exchange, but it does not guarantee that every dependency is reachable, fast, or authorized. Understanding the path makes failures easier to localize and policies safer to introduce.

Start with the Kubernetes network model

The Kubernetes networking model gives every Pod its own cluster-wide IP address. Containers in the same Pod share a network namespace and communicate over localhost. Pods normally reach other Pods directly, while Services give a changing set of Pods a stable virtual address and DNS name.

For an ML platform, those primitives usually map to four traffic classes:

Treating all four as “networking” hides different failure modes. A training Pod can reach the Kubernetes API while failing to read its dataset. A model Service can pass DNS resolution but still have no ready endpoints.

Follow the complete data path

When a workload cannot reach a dependency, inspect the path in order:

1. Confirm the workload is running in the intended cluster and namespace.
2. Resolve the destination name from the workload's network context.
3. Check that the Service has the expected selector, port, and ready endpoints.
4. Check ingress, egress, firewall, proxy, and NetworkPolicy rules.
5. Verify application-level authentication and TLS separately from connectivity.
6. Measure latency and throughput from the workload, not from an operator's laptop.

This sequence prevents a common debugging mistake: changing DNS, credentials, and policy at the same time without identifying which layer failed.

Polyaxon connections keep external systems such as artifact stores, registries, and data services explicit in workload configuration. That makes it easier to inventory required destinations before tightening network policy.

Use Services for stable discovery

A Pod IP is temporary. A Kubernetes Service selects Pods and presents a stable name and address. Inside the same namespace, a workload can usually use the short Service name. Across namespaces, prefer a fully qualified name such as:

model-router.inference.svc.cluster.local

DNS success does not prove that traffic can be served. Inspect the Service, EndpointSlices, and readiness state together:

kubectl --context production --namespace inference get service model-router
kubectl --context production --namespace inference get endpointslice \
  --selector kubernetes.io/service-name=model-router
kubectl --context production --namespace inference get pods \
  --selector app=model-router

If the Service exists but has no endpoints, the problem is usually a selector mismatch or Pods that are not ready—not the cluster DNS service.

Separate ingress from egress

Ingress brings requests into a cluster. Egress lets workloads reach systems outside it. ML systems need both, but their policies are rarely symmetrical.

A model endpoint might accept inbound traffic only through a Gateway or ingress controller. The same Pod may need outbound access to a model registry and telemetry backend. Training jobs may need no inbound traffic at all while requiring high-throughput access to object storage.

Document allowed flows as source, destination, port, protocol, and purpose. Avoid broad rules such as “allow the ML namespace to the internet.” A precise dependency inventory gives security teams something reviewable and gives operators a checklist when a destination changes.

Introduce NetworkPolicy with evidence

NetworkPolicy controls which Pod traffic is allowed, provided the cluster's network plugin enforces it. Start by observing real dependencies, then introduce policy in small steps.

Useful safeguards include:

• preserve DNS access while applying default-deny egress;
• select both sources and destinations with stable labels;
• account for artifact stores, package indexes, identity endpoints, and telemetry;
• test distributed worker communication before enforcing the policy;
• maintain an operational path for debugging without permanently broad access.

Our Kubernetes network policy guide covers the policy mechanics. The important design step comes first: know which flows the workload actually needs.

Observe the network as part of the run

CPU and GPU charts cannot explain time lost to slow artifact downloads, DNS retries, or blocked peer traffic. Capture network evidence alongside workload metadata:

• DNS error and latency rates;
• connection failures and resets;
• request latency by destination;
• bytes transferred for datasets and artifacts;
• distributed communication time;
• Service endpoint changes;
• policy-denied flows, where the network implementation exposes them.

Correlate that evidence with the job, replica, node, dataset version, and execution window. Polyaxon's platform observability provides the workload context; cluster and network telemetry complete the explanation.

The goal is not to make ML engineers become network specialists. It is to make dependencies explicit enough that a failed run can be traced to the right layer without guesswork.