Aug 20, 2024

Kubernetes network policies: an actionable guide

Use Kubernetes network policies to control pod traffic, restrict lateral movement, and make cluster networking less permissive by default.

Kubernetes networking is permissive unless you make it stricter. In many clusters, pods can talk to far more things than they need. Network policies give teams a way to narrow that surface.

For ML workloads, that matters because jobs often touch data stores, artifact stores, registries, and internal APIs. A compromised or misconfigured pod should not get a free tour of the cluster.

Why do you need network policies?

By default, all pods in Kubernetes are non-isolated, meaning that all pods are able to talk to one another. This is true for pods in the same namespace and pods in different namespaces. While this is fine for most organizations, there are many good reasons to consider implementing network policies.

As mentioned earlier in this article, network policies allow engineers to restrict communication access between services. The most important reason to implement network policies has to do with the principle of zero trust: the concept that no service or network can be trusted, including your own services and networks. As an example, on a day-to-day basis, there would probably be no problem with your frontend being able to access your database. Most engineers would agree that the frontend should never access the database - but if the ability exists, it's not too big of an issue as long as you don't implement any access functionality. However, even a scenario like this leaves a door open for hackers.

Envision an infrastructure with three services: a frontend, a backend, and a database. With an open network, a hacker needs only to get access to any service, from which they can then access the database. If you lock down network access so that only the backend can access the database, it's no longer enough for the hacker to get access to the frontend service - they need to get access to the backend if they want access to the database. This is one of the most important reasons for implementing strict network policies.

How do network policies work?

The first thing to know about network policies is that they require a network plug-in. There are two types of network plug-ins in Kubernetes: CNI and kubenet. One of the many available CNI plug-ins will work with a network policy. If you're using a managed service from a cloud provider, it's also worth determining whether they provide specific ways of enabling network plug-ins. For example, this is how you enable them in Google Kubernetes Engine.

Once you've got the network plug-in set up, you can define your network policies. They're defined just like any other resource you work with in Kubernetes, and they require standard fields such as apiVersion, kind, metadata, and spec. Next, you either define one of the two forms of isolation possible in Kubernetes or specify both, as ingress and egress can be defined together or separately. It's worth noting that even though they may be specified in the same file, they're still referred to as separate forms of isolation.

Once you've defined your manifest file, it's a simple matter of using kubectl apply to apply it. When creating your network policy, you can choose your ingress selectors in three different ways: CIDR (selection of IPs), podSelector, and namespaceSelector. This gives you very granular control over where traffic is allowed to come from.

It's important to note that network policies are additive, not sequential - all of them will be applied, and newer ones don't supersede older ones. So you don't need to worry about the order you create your network policies in, but you do need to be sure that you don't have conflicting network policies, as that can lead to confusing troubleshooting.

One last thing to know about network policies is that they have to be mutual. If one service allows communication but the service on the other end doesn't, the connection will be dropped.

Implementing a network policy

To get a solid idea of how network policies can be implemented, it's best to see an example.

First, you need to determine how to enable the network plug-in in your cluster and enable it. Once it's enabled, you need to create the services used in this example: a frontend, a backend, and a database. To make this easier, all three services will be represented by an Nginx server, as that allows you to easily test the connection.

Start by creating a file called frontend.yaml and paste the following into it:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend-deployment
  labels:
    app: frontend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 80
          lifecycle:
            postStart:
              exec:
                command: [ "/bin/bash", "-c", "apt update && apt install -y curl" ]
---
apiVersion: v1
kind: Service
metadata:
  name: frontend
spec:
  selector:
    app: frontend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

This is a simple YAML manifest file specifying two distinct resources. The first is a deployment that spins up three instances of an Nginx service. Next, a service is defined, allowing the pods to easily communicate with one another. In the spec field of the deployment, you'll notice the field lifecycle.postStart.exec.command, which installs curl. This is because you'll be using curl to test the connection between services and verify that the network policies are working.

Save this file in an empty directory, and create two copies of it. Name one of them backend.yaml, and change all occurrences of frontend to backend. Name the other database.yaml, and change all occurrences of frontend to database. Now all the manifest files are ready, and you deploy them by executing kubectl apply -f . in your terminal. Within a few seconds, you'll have all the services spun up; you can verify this by running kubectl get pods.

Once they're all running, get a shell inside the frontend service by running kubectl exec -it svc/frontend bash. With the shell open, run curl backend. You should see that the pod is able to communicate with the backend, and it can also communicate with the database by running curl database. This is the expected behavior, but now you want to lock down the frontend's access to the database.

To do this, save the following to a file called access-backend-database.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: no-access-frontend-database
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: database
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: backend

This policy will be deployed in the default namespace, and the rule will match any pods with the label app: database. The rule specifies that any ingress from pods with the label app: backend will be allowed. You might notice that the frontend isn't mentioned anywhere here - this is because network policies are all about specifying the traffic that is allowed. Any traffic not specified will be dropped. This is also why this rule is being applied to the database rather than the frontend.

Applying a rule to the frontend specifying that it's not allowed to access the database will work, and the frontend won't be able to access the database. But what happens when you spin up another service that shouldn't be allowed to access the database? Then you need to create a new policy. Therefore, it's better to tell the database what services are allowed to access it, as any other service will automatically be denied.

Now test this by executing kubectl apply -f access-backend-database.yaml. This will apply the network policy. Enter the frontend shell again by running kubectl exec -it svc/frontend bash, and run curl database. After a while, the curl request will time out, demonstrating that the access is now being denied.

This is a simple example of how a network policy that restricts traffic between services can be implemented. You can make stricter policies with namespace selectors and IP ranges. Policies can be used to lock down your entire cluster, ensuring that only approved paths can be used between services.

Further options

What you've seen in the previous section is an example of a simple network policy. You can get more specific with selectors such as namespaceSelector and ipBlock. Beyond this, network policies don't offer a lot of advanced use cases. While getting network policies to work can be a complicated task, there aren't a lot of options to use when you're configuring them.

The only missing option you need to learn about at this point is how you can target a range of ports. This is done by adding a field endPort to your port specification. For example:

ingress:
  - from:
      - podSelector:
          matchLabels:
            app: backend
    ports:
      - protocol: TCP
        port: 80
        endPort: 8088

Conclusion

Network policies are a practical way to reduce unnecessary pod-to-pod and pod-to-service access. Start with the traffic the workload actually needs, then deny the rest.

For ML workloads, that means being explicit about data stores, artifact stores, registries, tracking APIs, and service endpoints. Polyaxon centralizes many of those connections, but the cluster network still needs real boundaries.