I need to disable all egress traffic from my pod in namespace, except one direction, for example yahho.com My service.yaml looks like:
kind: Service
apiVersion: v1
metadata:
name: yahoo
labels:
output: allow
spec:
type: ExternalName
externalName: yahoo.com
And my network policy file blocks all output traffic:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: np-test
spec:
podSelector: {}
policyTypes:
- Egress
I've try to use the construction for allow:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-yahoo
namespace: np-test
spec:
policyTypes:
- Egress
podSelector:
matchLabels:
name: test-ubuntu
egress:
- to:
- namespaceSelector:
matchLabels:
namespace: np-test
- podSelector:
matchLabels:
output: allow
ports:
- protocol: TCP
port: 443
Can you correct my construction podSelector for passing traffic? I can't find any ideas in documentation.
Unfortunately GKE does not support Network Policies with DNS-name-based egress filtering "out-of-the-box" due to Calico limitations: CIDR-based filtering is inflexible and error-prone, and does not work well for external resources. There is a Feature Request on GitHub dated by 2017 for introducing support for a new parameter DNSSelector
in Kubernetes API.
As for now, a third-party solution, the Cilium CNI plug-in provides DNS-based egress filtering. What is important, it supports wildcards.
Steps in brief:
kubenet
).As soon as you create the first network policy in a namespace, all other traffic is denied. There's no need in a headless service of type ExternalName
but if you preferred to use it, restricted access via such service would work as well. kube-dns
is in charge of name resolution. The network policy grants access to it.
Steps in details:
$ gcloud services enable --project "mylab" container.googleapis.com
kubenet
). A parameter --username
causes Basic authentication to be enabled.$ gcloud container --project "mylab" clusters create "standard-cluster-1" --username "admin" --zone "europe-west3-c"
$ gcloud auth list
$ gcloud auth login
$ gcloud container clusters get-credentials standard-cluster-1 --zone europe-west3-c --project mylab
$ kubectl config get-clusters
$ kubectl cluster-info`
$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user youraccount@google.com
np-test
for your workloads and label it:$ kubectl create namespace np-test
$ kubectl label namespace/np-test namespace=np-test
$ kubectl get namespace/np-test --show-labels`
example.com
, if you need.$ vi example-service.yaml
apiVersion: v1
kind: Service
metadata:
name: example-service
namespace: np-test
spec:
type: ExternalName
externalName: example.com
$ kubectl apply -f example-service.yaml
$ kubectl get services -n np-test
$ vi busybox-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox-pod
namespace: np-test
labels:
namespace: np-test
spec:
containers:
- image: radial/busyboxplus:curl
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
name: busybox-container
restartPolicy: Always
$ kubectl create -f ./busybox-pod.yaml
$ kubectl get pods -n np-test --show-labels
$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh
$ helm version
$ curl -LO https://github.com/cilium/cilium/archive/1.6.5.tar.gz
$ tar xzvf 1.6.5.tar.gz
$ cd cilium-1.6.5/install/kubernetes
Generate the required YAML files:
$ helm template cilium \
--namespace cilium \
--set global.nodeinit.enabled=true \
--set nodeinit.reconfigureKubelet=true \
--set nodeinit.removeCbrBridge=true \
--set global.cni.binPath=/home/kubernetes/bin \
> cilium.yaml
$ kubectl create namespace cilium
$ kubectl get namespaces
$ kubectl create -f cilium.yaml
Restart all pods in the kube-system
namespace so they can be managed by Cilium:
$ kubectl delete pods -n kube-system $(kubectl get pods -n kube-system -o custom-columns=NAME:.metadata.name,HOSTNETWORK:.spec.hostNetwork --no-headers=true | grep '<none>' | awk '{ print $1 }')
$ kubectl -n cilium get pods
$ kubectl exec -n np-test busybox-pod -- curl -kL google.com
$ kubectl exec -n np-test busybox-pod -- curl -kL yahoo.com
$ kubectl exec -n np-test busybox-pod -- curl -kL example.com
$ kubectl exec -n np-test busybox-pod -- curl -kL -H 'host: example.com' example-service
$ kubectl exec -n np-test busybox-pod -- nslookup example.com
$ kubectl exec -n np-test busybox-pod -- nslookup example-service
Note that example-service
is resolved into the same IP as example.com
example.com
only.$ vi dns-policy.yaml
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: "dns-policy"
namespace: np-test
spec:
endpointSelector:
matchLabels:
namespace: np-test
egress:
- toFQDNs:
- matchName: "example.com"
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": kube-system
"k8s:k8s-app": kube-dns
toPorts:
- ports:
- port: "53"
protocol: ANY
rules:
dns:
- matchPattern: "*"
$ kubectl create -f dns-policy.yaml
$ kubectl get CiliumNetworkPolicy -n np-test
$ kubectl exec -n np-test busybox-pod -- ping -c 3 google.com
100% packet loss
$ kubectl exec -n np-test busybox-pod -- ping -c 3 yahoo.com
100% packet loss
$ kubectl exec -n np-test busybox-pod -- ping -c 3 example-service
0% packet loss
$ kubectl exec -n np-test busybox-pod -- ping -c 3 example.com
0% packet loss
$ kubectl exec -n np-test busybox-pod -- curl -kL google.com
Connection timed out
$ kubectl exec -n np-test busybox-pod -- curl -kL yahoo.com
Connection timed out
$ kubectl exec -n np-test busybox-pod -- curl -kL example.com
OK
$ kubectl exec -n np-test busybox-pod -- curl -kL -H 'host: example.com' example-service
OK
As you can see the Cilium network policy allows access to the example.com
directly or via a headless service whereas access to other sites is restricted.
This approach allows to obtain DNS-name-based egress filtering on GKE.
The links below provide more details regards described solution:
Kubernetes Service: Headless Services
Kubernetes Service: Type ExternalName
GKE: Creating a cluster network policy
NetworkPolicy Egress with CIDRSelector and DNSSelector #50453
Kubernetes: Declare Network Policy
Open Source DNS-aware Kubernetes Network Policies Using Cilium
Installation of Cilium on Google GKE
Cilium: Locking down external access with DNS-based policies