Polyaxon v3 is coming →

Protect EKS workloads from instance metadata risk

Prevent compromised Amazon EKS Pods from inheriting node credentials by using workload identity, restricting EC2 instance metadata, and detecting access attempts.

September 15, 2020by Polyaxon
Protect EKS workloads from instance metadata risk

Amazon EC2 Instance Metadata Service (IMDS) provides information and temporary credentials to software running on an instance. On Amazon EKS worker nodes, that can become a privilege path: a compromised Pod may query IMDS and receive the node instance profile unless access is deliberately restricted.

The durable fix is to separate workload identity from node identity. Each application should receive only the AWS permissions it needs, while ordinary Pods should be unable to use the worker node's role.

Understand the credential path

The EKS node role is designed for node-level responsibilities such as joining the cluster, pulling images, or operating required add-ons. If a general workload can obtain those credentials through IMDS, an application vulnerability can become an AWS account incident.

Risk depends on three questions:

  1. Can the Pod reach IMDS?
  2. What permissions does the node role have?
  3. Can the attacker use those credentials outside the node or cluster?

Do not assume a network namespace automatically blocks metadata access. Pods using hostNetwork: true can access IMDS, and ordinary Pods may be able to reach it when node metadata options and network controls permit the path.

Give every workload a dedicated identity

Use EKS Pod Identity or IAM roles for service accounts (IRSA) to associate an IAM role with a dedicated Kubernetes ServiceAccount. Grant one application role per distinct permission set rather than sharing a broad role across a namespace.

The IAM policy should name the required actions and resources. A training workload that reads one S3 prefix does not need the ability to modify node groups, inspect unrelated buckets, or pass IAM roles. Separate development and production roles even when the application code is identical.

Workload identity improves attribution as well as containment. AWS API events can identify the workload role instead of collapsing activity into a shared node credential.

Restrict access to IMDS

AWS documents the relevant settings in the EKS identity and access management best practices. Where compatible with the node image, CNI, and required add-ons, require IMDSv2 and set the response hop limit so ordinary Pods cannot retrieve node credentials.

Roll this out as an engineering change, not a blind toggle. Inventory software that legitimately uses instance metadata, including bootstrap logic, networking components, storage drivers, monitoring agents, and autoscaling tools. Migrate those components to supported identity mechanisms or narrowly controlled node access before enforcing the restriction.

Use node groups or launch templates to keep metadata settings consistent. Detect nodes that drift from the approved configuration.

Restrict host networking

Pods with host networking share the node's network namespace and retain access to IMDS even when the hop limit blocks ordinary Pod networking. Deny hostNetwork, hostPID, and hostIPC by default through Pod Security Admission or an admission policy, then approve only documented system workloads that require them.

Run privileged infrastructure components in dedicated namespaces with tightly controlled RBAC. If a third-party DaemonSet requests host networking, understand why, identify the AWS permissions it can reach, and review upgrades for changes to that requirement.

Minimize the node role

Metadata restrictions reduce reachability, but the node role still needs least privilege. Remove application permissions and unrelated administrative actions. Keep node, image-pull, and add-on permissions separate where the EKS architecture supports it.

Review the role as capabilities move to Pod Identity or IRSA. A permission that was necessary during migration should not remain indefinitely. Use separate roles for specialized node pools so a compromised general-purpose worker cannot inherit GPU, data, or production privileges it never needs.

Detect access and credential misuse

Monitor both the metadata path and the credentials' effects. Useful signals include unexpected requests to the link-local metadata address, AWS API calls made with the node role from unusual services or regions, denied calls following a successful metadata request, and new Pods that enable host networking.

Correlate CloudTrail events with Kubernetes audit logs, Pod identity, namespace, node, image, and workload owner. Preserve evidence before replacing a suspicious Pod or node. A deleted Pod should not erase the connection between an AWS API call and the operation that initiated it.

Roll out safely

Start with an inventory of node roles, ServiceAccounts, host-networked Pods, and metadata-dependent components. Introduce workload identities, reduce node permissions, enforce metadata settings on a canary node group, and verify node bootstrap, DNS, networking, storage, monitoring, and autoscaling before expanding.

Polyaxon operations can use dedicated ServiceAccounts and approved connections so training jobs, services, notebooks, and automation do not share node credentials. Record the workload identity and required data access alongside the component definition, then validate the deployed Pod and AWS role as part of release policy.