Slow network on AKS ith Egress policies

2/8/2022

I've setup a Kube cluster with AKS, Calico Policy and Azure CNI plugin.

Some of my pods need to connect to external services and I want to setup Egress rules to limit traffic on the pods. Ingress is doing just fine but for whatever reason when I add my Egress rules it works but it seems to be painfully slow.

For example : without Egress my logs telling me my pod is connected to the DB are instant. With egress rules up, it can takes 3 to 10 minutes to connect to my DB.

Before connecting to the DB my pod is getting values from a KeyVault. (I don't use Vault mounting because the variables fetched in the vault are dynamic depending on a certain configuration). So I'm not sure what takes times here.

I've done a tcpdump and ifconfig but I can't see any dropped packets.

I haven't set ports on CIDR yet because I wanted to make sure it was not a matter of port mapping first.

The pod runs on port 3000 and here are my configs for the network policies :

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  namespace: client2
  name: default-deny
spec:
  podSelector:
    matchLabels: {}
  policyTypes:
    - Ingress
    - Egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: client2-network-policy
  namespace: client2
spec:
  podSelector:
    matchLabels:
      io.kompose.service: backend
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            name: ingress-basic
        podSelector:
          matchLabels:
            app.kubernetes.io/name: ingress-nginx
            app.kubernetes.io/component: controller
            app.kubernetes.io/instance: nginx-ingress
      ports:
        - protocol: TCP
          port: 3000
    - from:
      - namespaceSelector:
          matchLabels:
            networking/namespace: kube-system
        podSelector:
          matchLabels:
            k8s-app: kube-dns
      ports:
        - protocol: TCP
          port: 53
        - protocol: UDP
          port: 53
    - from:  # Keyvault ips fetched from dns lookup
      - ipBlock:
          cidr: 1.2.3.4/32
    - from:
      - ipBlock:
          cidr: 3.4.5.6/32
    - from:
      - ipBlock:
          cidr: 34.5.4.6/32
    - from:
      - ipBlock:
          cidr: 2.5.6.2/32 #db shard1
    - from:
      - ipBlock:
          cidr: 2.4.5.2/32 #db shard2
    - from:
      - ipBlock:
          cidr: 1.2.4.3/32 #db shard3
  egress:
    - to:
      - namespaceSelector:
          matchLabels:
            name: ingress-basic
        podSelector:
          matchLabels:
            app.kubernetes.io/name: ingress-nginx
            app.kubernetes.io/component: controller
            app.kubernetes.io/instance: nginx-ingress
    - to:
        - namespaceSelector:
            matchLabels:
              networking/namespace: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: TCP
          port: 53
        - protocol: UDP
          port: 53
    - to:
      - ipBlock:
          cidr: 1.3.8.0/24 #unknown azure service i saw in tcpdump
    - to:  # Keyvault ips fetched from dns lookup
      - ipBlock:
          cidr: 1.2.3.4/32
    - to:
      - ipBlock:
          cidr: 3.4.5.6/32
    - to:
      - ipBlock:
          cidr: 34.5.4.6/32
    - to:
      - ipBlock:
          cidr: 2.5.6.2/32 #db shard1
    - to:
      - ipBlock:
          cidr: 2.4.5.2/32 #db shard2
    - to:
      - ipBlock:
          cidr: 1.2.4.3/32 #db shard3

I don't know where to look at to debug the slowness, any ideas or tips ? Thanks !

-- Thr
azure
azure-aks
kubernetes

0 Answers