We are upgrading our AKS cluster in order to use Standard SKU load balancers that went GA recently. See this Microsoft Update Notification.Previously only basic SKU load balancers were available and they would not allow us to send a TCP reset when connections went stale. This lead to a lot of creative work arounds to deal with stale connections in connection pools for example.
So during creation of an ingress I can configur the load balancer by using annontations. For example I can set type to internal and timeout settings using annotations. However being able to set the TCP reset flag to true via annotations does not seem possible. I have found with some digging a list of the annotations in this Go Walker page.
I have managed to create a ingress controller using the following yaml. Note the annonations.
controller:
service:
loadBalancerIP: 172.15.23.100
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout: "15"
I ran the following commands:
helm install stable/nginx-ingress --namespace ingress -f dev-ingress.yaml --name dev-ingress --set controller.replicaCount=3
After a minute or so I can see the internal loadbalancer getting the specified IP address and I can also see it on the console see below:
kubectl -n ingress get svc dev-ingress-nginx-ingress-controller
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
dev-ingress-nginx-ingress-controller LoadBalancer 172.15.24.11 172.15.23.100 80:30962/TCP,443:30364/TCP 24m app=nginx-ingress,component=controller,release=dev-ingress
However the load balancing rules are created with a TCP reset to false. Which requires me to log into the console and change it. See screen shot below:
I really would like to script this into the creation as doing things via interfaces leads to Snowflake deployments.
Something like the yaml below:
controller:
service:
loadBalancerIP: 172.15.23.100
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout: "15"
service.beta.kubernetes.io/azure-load-balancer-tcp-reset: "true"
Anyone know how I can configure this during service/ingress creation?
Update:
Based on the limitations documented on the TCP Reset setting for loadbalancers document it appears that it is not supported from kubectl. However it also says that the portal is not supported.
You can take a look at Cloud provider for Azure. It provides an annotation to set the TCP reset of the load balancer rules, but it's only available for version 1.16 or later and the latest version for AKS is 1.15.
You can use aks-engine to achieve your purpose if you really want to use it. The aks-engine already supports version 1.16 for Kubernetes. Remember, create the aks-engine cluster with the standard load balancer.
seeing this file does not have such an annotation I would conclude this is not yet possible with annotations. you'd have to figure some other way, or create a pull request to kubernetes to support such an annotation