I stumbled upon this strange limitation when deploying more than 3 applications utilizing AWS ALB within an ingress controller.
Up to 3 Applications, there's no problem, when a 4th ingress is provisioned, it doesn't get an endpoint:
However the LoadBalancer (ALB) is provisioned as expected:
Is there a limitation of 3 Ingresses I'm not aware of? Or did I set this limit somewhere in the configuration?
Output of kubectl describe ingress
of "failing" ingress (note the missing address - the other 3 ingresses all have addreses):
Name: some-ingress
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
*
/ ssl-redirect-default:use-annotation (<none>)
/* ssl-redirect:use-annotation (<none>)
/* some-service:80 (192.168.92.252:8080)
Annotations:
alb.ingress.kubernetes.io/actions.ssl-redirect: {"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}
alb.ingress.kubernetes.io/actions.ssl-redirect-default: {"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Path": "/someapp/#{path}", "Port": "443", "StatusCode": "HTTP_301"}}
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-central-1:...
alb.ingress.kubernetes.io/listen-ports: [{"HTTP": 80}, {"HTTPS":443}]
alb.ingress.kubernetes.io/scheme: internet-facing
kubernetes.io/ingress.class: alb
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 16m alb-ingress-controller LoadBalancer some-alb created, ARN: some-alb-arn:loadbalancer/app/some-alb/some-ids
Normal CREATE 16m (x2 over 16m) alb-ingress-controller rule 1 created with conditions [{ Field: "path-pattern", Values: ["/"] }]
Normal CREATE 16m (x2 over 16m) alb-ingress-controller rule 2 created with conditions [{ Field: "path-pattern", Values: ["/*"] }]
Normal CREATE 16m alb-ingress-controller rule 3 created with conditions [{ Field: "path-pattern", Values: ["/*"] }]
Found the problem, apparently there is a Security Group Limit of 5 per Cluster Node ENI (Elastic Network Interface), and every new application instance with ALB Ingress adds a SG. This is a service quota, which can be increased:
The relevant logs were found via:
kubectl logs -n kube-system deployment.apps/alb-ingress-controller
I already had 2 SGs existent for the ENIs, so after 3 application instances were deployed, the limit was reached. I just increased the limit to 15 (per service quota request) and voilĂ , I can deploy more application instances with ALB Ingresses.
For me, this solved the issue, but this solution will not scale endlessly, as with every instance of an application a new Security Group is added to the ENIs, so 13 application instances is the new limit for me.