Detect malicious activity in Amazon EKS
Detect and contain suspicious Amazon EKS behavior by correlating Kubernetes audit, AWS API, DNS, network, runtime, and workload context.
An Amazon EKS incident rarely appears in one log. A vulnerable public service may lead to command execution inside a Pod, DNS discovery, connections to internal services, access to node metadata, use of AWS credentials, and Kubernetes API changes. Each signal looks incomplete in isolation.
Effective detection connects activity across the EKS control plane, AWS account, network, node runtime, and the application or ML operation that owns the Pod.
Start with likely attack paths
Model how an attacker could move from a reachable workload to valuable data or control. Include public endpoints, interactive notebooks, uploaded code, images, model artifacts, ServiceAccount tokens, cloud workload identities, node credentials, storage systems, and the Kubernetes API.
Prioritize detections for behavior that crosses a trust boundary:
- a public service begins scanning cluster addresses;
- a workload contacts EC2 instance metadata;
- a Pod reads Secrets or cluster-scoped resources for the first time;
- a new privileged, host-networked, or host-mounted Pod appears;
- an application identity calls unexpected AWS services;
- a workload reaches an unusual external domain or transfers abnormal volumes;
- an operator disables logging, policy, or security tooling.
A threat model keeps the signal program tied to credible impact instead of collecting every available event.
Collect evidence from every layer
Enable the EKS control-plane log categories required by the operating model, especially Kubernetes API and audit evidence. Retain AWS CloudTrail for cluster, IAM, networking, compute, and data-service activity. AWS summarizes the available sources in its EKS monitoring guidance.
Add DNS queries, VPC and cluster network flows, load-balancer or gateway access logs, node and container runtime events, admission decisions, image provenance, and application logs. Monitor the collectors for dropped events, failed exports, queue growth, and configuration changes.
Normalize cluster, account, region, namespace, ServiceAccount, Pod, node, image digest, source address, and workload owner. Preserve unique operation IDs in logs or traces rather than turning them into high-cardinality metric labels.
Establish behavior baselines
Baseline by workload class rather than across the entire cluster. A controller that watches many objects, an interactive notebook, a batch trainer, and an inference service have different API, DNS, network, and process patterns.
Document expected destinations, AWS actions, process entry points, data volume, execution duration, and resource shape. Version the baseline with the application. A legitimate release that adds a dependency should update the expected path through review rather than silently training the detector to accept it.
Use deterministic policy for conditions that should never occur, such as ordinary training jobs creating cluster roles. Use anomaly detection for variable behaviors such as destination novelty or unexpected data transfer.
Correlate identities and timelines
Translate an AWS API call back to the IAM role and the Kubernetes ServiceAccount or node that received it. Translate a network connection back to a Pod and then to the Deployment, Job, notebook, or Polyaxon run that created the Pod.
Time synchronization and retention are essential. Short-lived Pods disappear quickly, while cloud events may arrive later. Preserve object metadata and ownership relationships long enough to reconstruct the timeline without keeping sensitive payloads indefinitely.
Group multiple symptoms into one incident. A shell process, new DNS destination, metadata request, and denied S3 call from the same workload should produce a coherent investigation, not four unrelated queues.
Triage without destroying evidence
Confirm the workload identity, image digest, command, owner, start time, recent deployment change, network destinations, Kubernetes API activity, and AWS calls. Determine whether the activity is expected automation, a misconfiguration, or an active compromise.
Avoid immediately deleting the only Pod that contains volatile evidence. First preserve the relevant logs, object definition, runtime events, identities, and network timeline according to the incident procedure. Do not run ad hoc investigation commands inside a compromised container if they could execute attacker-controlled tools or alter evidence.
Contain at multiple boundaries
Containment may include stopping the controller that recreates the Pod, isolating a namespace or node, applying emergency network restrictions, blocking a destination, revoking Kubernetes and AWS credentials, quarantining an image or artifact, and denying new admissions from the affected source.
Choose actions based on the suspected authority. Deleting one Pod does not help when the controller, image tag, stolen role, or admission path remains compromised. Rebuild nodes from a trusted image when node-level access is plausible.
Document the recovery criteria: trusted image and artifacts, patched entry point, rotated credentials, restored controls, clean nodes, validated network paths, and enhanced detections.
Connect security to ML operations
ML workloads add high-value data, expensive accelerators, user-supplied code, long-running jobs, and model files that may execute during loading. Security teams need the infrastructure evidence, while ML platform teams need to know which experiments, models, and downstream deployments are affected.
Polyaxon tracks code, parameters, inputs, images, environments, artifacts, projects, owners, and run lineage. Correlating that context with EKS and AWS telemetry turns an ephemeral Pod alert into an actionable answer: what ran, who owned it, which data it reached, what it produced, and what else requires review.