GKE Network policy and ExternalName

12/21/2019

I need to disable all egress traffic from my pod in namespace, except one direction, for example yahho.com My service.yaml looks like:

kind: Service
apiVersion: v1
metadata:
 name: yahoo
 labels:
   output: allow
spec:
 type: ExternalName
 externalName: yahoo.com

And my network policy file blocks all output traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: np-test
spec:
  podSelector: {}
  policyTypes:
  - Egress

I've try to use the construction for allow:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-yahoo
  namespace: np-test
spec:
  policyTypes:
  - Egress
  podSelector:
    matchLabels:
      name: test-ubuntu
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          namespace: np-test 
    - podSelector:
        matchLabels:
          output: allow
    ports:
    - protocol: TCP
      port: 443

Can you correct my construction podSelector for passing traffic? I can't find any ideas in documentation.

-- Aleksandr Khomenko
google-kubernetes-engine
kubernetes-networkpolicy

1 Answer

1/10/2020

Unfortunately GKE does not support Network Policies with DNS-name-based egress filtering "out-of-the-box" due to Calico limitations: CIDR-based filtering is inflexible and error-prone, and does not work well for external resources. There is a Feature Request on GitHub dated by 2017 for introducing support for a new parameter DNSSelector in Kubernetes API.

As for now, a third-party solution, the Cilium CNI plug-in provides DNS-based egress filtering. What is important, it supports wildcards.

Steps in brief:

  1. Enable the GKE API for the current project, if not has been already done.
  2. Build a standard 3-node GKE cluster without Network Policy support (with kubenet).
  3. Connect to the freshly created Kubernetes cluster.
  4. Create a cluster-admin-binding in GKE.
  5. Create a custom namespace for your workloads, if necessary.
  6. Create a headless service for the external resource, if necessary.
  7. Run a Busybox container for checking the network policy effect.
  8. Install Helm.
  9. Use Helm to deploy Cilium.
  10. Check availability of external resources before applying a network policy.
  11. Create and deploy a Cilium network policy.
  12. Check availability of external resources after applying a network policy.

As soon as you create the first network policy in a namespace, all other traffic is denied. There's no need in a headless service of type ExternalName but if you preferred to use it, restricted access via such service would work as well. kube-dns is in charge of name resolution. The network policy grants access to it.

Steps in details:

  1. Enable the GKE API for the current project, if not enabled yet.
$ gcloud services enable --project "mylab" container.googleapis.com
  1. Build a standard 3-node GKE cluster without Network Policy support (with kubenet). A parameter --username causes Basic authentication to be enabled.
$ gcloud container --project "mylab" clusters create "standard-cluster-1" --username "admin" --zone "europe-west3-c"
  1. Connect to the freshly created Kubernetes cluster
$ gcloud auth list
$ gcloud auth login
$ gcloud container clusters get-credentials standard-cluster-1 --zone europe-west3-c --project mylab
$ kubectl config get-clusters
$ kubectl cluster-info`
  1. Create a cluster-admin-binding in GKE.
$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user youraccount@google.com
  1. Create a custom namespace np-test for your workloads and label it:
$ kubectl create namespace np-test
$ kubectl label namespace/np-test namespace=np-test
$ kubectl get namespace/np-test --show-labels`
  1. Create a headless service for the external resource example.com, if you need.
$ vi example-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: example-service
  namespace: np-test
spec:
  type: ExternalName
  externalName: example.com

$ kubectl apply -f example-service.yaml
$ kubectl get services -n np-test
  1. Run a Busybox container for checking the network policy effect.
$ vi busybox-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: busybox-pod
  namespace: np-test
  labels: 
    namespace: np-test
spec:
  containers:
  - image: radial/busyboxplus:curl
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox-container
  restartPolicy: Always

$ kubectl create -f ./busybox-pod.yaml
$ kubectl get pods -n np-test --show-labels
  1. Install Helm on the CloudShell instance or a workstation you're working on:
$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh
$ helm version
  1. Use Helm to deploy Cilium.
$ curl -LO https://github.com/cilium/cilium/archive/1.6.5.tar.gz
$ tar xzvf 1.6.5.tar.gz
$ cd cilium-1.6.5/install/kubernetes

Generate the required YAML files:

$ helm template cilium \
  --namespace cilium \
  --set global.nodeinit.enabled=true \
  --set nodeinit.reconfigureKubelet=true \
  --set nodeinit.removeCbrBridge=true \
  --set global.cni.binPath=/home/kubernetes/bin \
  > cilium.yaml
$ kubectl create namespace cilium
$ kubectl get namespaces
$ kubectl create -f cilium.yaml

Restart all pods in the kube-system namespace so they can be managed by Cilium:

$ kubectl delete pods -n kube-system $(kubectl get pods -n kube-system -o custom-columns=NAME:.metadata.name,HOSTNETWORK:.spec.hostNetwork --no-headers=true | grep '<none>' | awk '{ print $1 }') 
$ kubectl -n cilium get pods
  1. Check availability of external resources from within the Busybox pod before applying a network policy.
$ kubectl exec -n np-test busybox-pod -- curl -kL google.com
$ kubectl exec -n np-test busybox-pod -- curl -kL yahoo.com
$ kubectl exec -n np-test busybox-pod -- curl -kL example.com
$ kubectl exec -n np-test busybox-pod -- curl -kL -H 'host: example.com' example-service
$ kubectl exec -n np-test busybox-pod -- nslookup example.com
$ kubectl exec -n np-test busybox-pod -- nslookup example-service

Note that example-service is resolved into the same IP as example.com

  1. Create and deploy a Cilium network policy. Now it's time to restrict access from the Pod to the Internet, and allow access to the example.com only.
$ vi dns-policy.yaml

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "dns-policy"
  namespace: np-test
spec:
  endpointSelector:
    matchLabels: 
      namespace: np-test
  egress:
  - toFQDNs:
    - matchName: "example.com"  
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": kube-system
        "k8s:k8s-app": kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"

$ kubectl create -f dns-policy.yaml
$ kubectl get CiliumNetworkPolicy -n np-test
  1. Check the effect of the Network Policy. Check availability of external resources from within the Busybox pod after applying a network policy.
$ kubectl exec -n np-test busybox-pod -- ping -c 3 google.com
100% packet loss
$ kubectl exec -n np-test busybox-pod -- ping -c 3 yahoo.com
100% packet loss
$ kubectl exec -n np-test busybox-pod -- ping -c 3 example-service
0% packet loss
$ kubectl exec -n np-test busybox-pod -- ping -c 3 example.com
0% packet loss
$ kubectl exec -n np-test busybox-pod -- curl -kL google.com
Connection timed out
$ kubectl exec -n np-test busybox-pod -- curl -kL yahoo.com
Connection timed out
$ kubectl exec -n np-test busybox-pod -- curl -kL example.com
OK
$ kubectl exec -n np-test busybox-pod -- curl -kL -H 'host: example.com' example-service
OK

As you can see the Cilium network policy allows access to the example.com directly or via a headless service whereas access to other sites is restricted.

This approach allows to obtain DNS-name-based egress filtering on GKE.

The links below provide more details regards described solution:

GKE: Network overview

Kubernetes Service: Headless Services

Kubernetes Service: Type ExternalName

Kubernetes Service: DNS entry

GKE: Creating a cluster network policy

Kubernetes: Network Policies

NetworkPolicy Egress with CIDRSelector and DNSSelector #50453

Kubernetes: Declare Network Policy

Open Source DNS-aware Kubernetes Network Policies Using Cilium

Installation of Cilium on Google GKE

Install Helm

Cilium: Locking down external access with DNS-based policies

-- mebius99
Source: StackOverflow