Protect Kubernetes workloads from DDoS attacks

A distributed denial-of-service attack tries to exhaust a scarce resource: network capacity, connection state, ingress workers, application threads, database pools, CPU, memory, or an expensive downstream operation. Kubernetes can reschedule and scale workloads, but scaling alone does not distinguish legitimate demand from hostile traffic.

Protection needs layers outside and inside the cluster, plus an operating plan for the point where capacity or automation reaches its limit.

Model external and internal denial of service

External attacks may flood a public address, open large numbers of connections, send expensive HTTP requests, or repeatedly invoke costly inference paths. Internal denial of service can begin with a compromised or misconfigured Pod that scans services, overwhelms DNS, saturates storage, fills queues, or consumes shared node resources.

For every public and high-value internal service, identify:

• the normal request, connection, and bandwidth ranges;
• the first resource likely to saturate;
• upstream and downstream dependencies;
• safe degraded behavior;
• the owner who can change traffic policy or capacity;
• the cost ceiling for automated scaling.

An ML endpoint may have modest request volume but extreme GPU cost per request. Rate controls must reflect computational work, not only packets per second.

Stop volume upstream

Large network floods should be absorbed before they reach the cluster or consume its internet link. Use the cloud provider's DDoS protections, load-balancing edge, content-delivery network, DNS resilience, and web application firewall where they match the protocol.

Restrict cloud firewall rules to the expected front doors. Keep the Kubernetes API, node management paths, metrics endpoints, and administrative services off the public application path. Maintain an emergency contact and escalation procedure for upstream mitigation.

Edge controls should reject invalid protocols, methods, headers, paths, request sizes, and known malicious sources before the application performs expensive work.

Apply application-aware limits

Configure connection, request, concurrency, body-size, header, and timeout limits at the load balancer, Gateway, Ingress, and application layers. Use authenticated quotas for tenants or API clients when possible; source IP alone is an imperfect identity behind proxies and distributed clients.

Protect expensive operations separately. Limit concurrent model loads, GPU inference requests, export jobs, artifact downloads, and unbounded queries. Use queues and backpressure so a burst does not become simultaneous work across every replica.

Cache safe repeatable responses and reject work early when the service cannot meet its objective. Make retry guidance explicit to avoid clients amplifying an outage with synchronized retries.

Contain east-west abuse

Kubernetes NetworkPolicy can restrict which Pods may reach a service, DNS, the API, node-local endpoints, and external networks. Establish default-deny policies and allow only required paths for protected namespaces.

NetworkPolicy is an isolation control, not a complete DDoS system. It does not provide a global traffic scrubber or application-aware rate limiter, and behavior depends on the networking implementation. Combine it with service identity, application authorization, and limits at the receiving service.

Isolate public serving from training, notebooks, controllers, and platform services. A flooded public endpoint should not share every node, queue, database pool, or ServiceAccount with the control path used to recover it.

Bound resource consumption

Set CPU, memory, and ephemeral-storage requests and limits according to measured behavior. Use namespace quotas and limit ranges to prevent one workload from consuming the cluster. Apply connection and worker-pool bounds inside the application because a CPU limit alone does not protect a database or external API.

Autoscaling can preserve availability during legitimate demand, but hostile traffic can turn it into uncontrolled cost. Scale on signals connected to useful work, define maximum capacity, protect constrained dependencies, and alert before the ceiling is reached.

For GPU services, queue depth and request latency are often more meaningful than average GPU utilization. Reserve capacity for health checks, cancellation, and control operations so saturation does not eliminate the recovery path.

Detect an attack early

Observe traffic rate, new source distribution, connection state, rejected requests, latency, error mix, queue depth, resource saturation, autoscaling decisions, DNS behavior, and downstream pressure. Compare current behavior with workload-specific baselines and known releases.

Separate demand from useful outcomes. A large rise in requests with flat successful predictions, completed jobs, or authenticated users is different from genuine adoption. Monitor telemetry health so dropped flow logs or overloaded metrics collectors do not look like recovery.

Practice the response plan

Preapprove actions for tightening edge rules, lowering expensive-operation quotas, blocking destinations, isolating workloads, adding bounded capacity, disabling nonessential features, and switching to a degraded mode. Define who can make each change and how it will be rolled back.

During the incident, preserve a timeline of traffic, policy changes, scaling, costs, and application impact. Afterward, adjust the earliest effective layer rather than only adding more replicas. Test the runbook with controlled load that stays within an agreed safety boundary.

Polyaxon helps separate ML control-plane activity from the workloads it schedules and records each operation's owner, inputs, resources, and outcome. Pair that context with cloud edge protection, Kubernetes isolation, and application-level limits to keep abusive traffic from consuming the capacity reserved for trusted training and serving work.