AKS Standard Load Balancer TCP Reset Annotations

12/3/2019

We are upgrading our AKS cluster in order to use Standard SKU load balancers that went GA recently. See this Microsoft Update Notification.Previously only basic SKU load balancers were available and they would not allow us to send a TCP reset when connections went stale. This lead to a lot of creative work arounds to deal with stale connections in connection pools for example.

So during creation of an ingress I can configur the load balancer by using annontations. For example I can set type to internal and timeout settings using annotations. However being able to set the TCP reset flag to true via annotations does not seem possible. I have found with some digging a list of the annotations in this Go Walker page.

I have managed to create a ingress controller using the following yaml. Note the annonations.

controller:
  service:
    loadBalancerIP: 172.15.23.100
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout: "15"

I ran the following commands:

helm install stable/nginx-ingress --namespace ingress -f dev-ingress.yaml --name dev-ingress --set controller.replicaCount=3

After a minute or so I can see the internal loadbalancer getting the specified IP address and I can also see it on the console see below:

kubectl -n ingress get svc dev-ingress-nginx-ingress-controller
NAME                                         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE   SELECTOR
dev-ingress-nginx-ingress-controller         LoadBalancer   172.15.24.11   172.15.23.100   80:30962/TCP,443:30364/TCP   24m   app=nginx-ingress,component=controller,release=dev-ingress

However the load balancing rules are created with a TCP reset to false. Which requires me to log into the console and change it. See screen shot below:

DefaultTCPRule

I really would like to script this into the creation as doing things via interfaces leads to Snowflake deployments.

Something like the yaml below:

controller:
  service:
    loadBalancerIP: 172.15.23.100
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout: "15"
      service.beta.kubernetes.io/azure-load-balancer-tcp-reset: "true"

Anyone know how I can configure this during service/ingress creation?

Update:

Based on the limitations documented on the TCP Reset setting for loadbalancers document it appears that it is not supported from kubectl. However it also says that the portal is not supported.

-- Namphibian
azure-aks
kubernetes-ingress
nginx

2 Answers

12/6/2019

You can take a look at Cloud provider for Azure. It provides an annotation to set the TCP reset of the load balancer rules, but it's only available for version 1.16 or later and the latest version for AKS is 1.15.

enter image description here

You can use aks-engine to achieve your purpose if you really want to use it. The aks-engine already supports version 1.16 for Kubernetes. Remember, create the aks-engine cluster with the standard load balancer.

-- Charles Xu
Source: StackOverflow

12/3/2019

seeing this file does not have such an annotation I would conclude this is not yet possible with annotations. you'd have to figure some other way, or create a pull request to kubernetes to support such an annotation

-- 4c74356b41
Source: StackOverflow