I'd like to create a nginx ingress controller with AWS internal NLB, the requirement is fix the IP address of NLB endpoint, for example, currently the NLB dns of Nginx ingress service is abc.elb.eu-central-1.amazonaws.com which is resolved to ip address 192.168.1.10, if I delete and re-create nginx ingress controller, I want the NLB DNS must be the same as before. Having a look in kubernetes service annotation, I did not see any way to re-use existing NLB, however, I find out the annotation service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses in link, as far as I understand that it allow me to set ip address for NLB, but it not work as my expectation, everytime I re-created nginx controller, the ip address is difference, Below is K8s service yaml file.
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses: "10.136.103.251"
service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-00df069133b22"
labels:
helm.sh/chart: ingress-nginx-3.23.0
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.44.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
spec:
type: LoadBalancer
externalTrafficPolicy: LocalI know this requirement is werid, is it possible to do that?
If your Kubernetes cluster runs on a VPC with more than one subnet (which is probably the case), you must provide a private ip address for each subnet.
I installed the AWS Load balancer controller with the helm chart, then i installed the nginx ingress controller with this helm chart :
helm install nginx-ingress ingress-nginx/ingress-nginx --namespace nginx-ingress -f internal-ingress-values.yamlHere the content of internal-ingress-values.yaml
controller:
ingressClass: nginx
service:
enableHttp: false
enableHttps: true
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-scheme: internal
service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses: 10.136.103.251, 10.136.104.251
service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-00a1a7f9949aa0ba1, subnet-12ea9f1df24aa332c
ingressClassResource:
enabled: true
default: trueAccording to the documentation the service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses annotation length/order must match subnets
So, you must provide IP addresses and subnet in the same order (don't mismatch). If you take my example above, you must make sure that :
It's a good idea to tag your subnets according to the documentation :
Key: kubernetes.io/cluster/my-cluster-name Value: shared
Key: kubernetes.io/role/internal-elb Value: 1
I tested this K8S on 1.20 and it works for my project. Don't provide "ingressClassResource" if you're on K8S <= 1.17.
The only LBs that will be managed (at least at the current version 2.3 of the AWS LB Controller) are "nlb-ip" and "external" types. This is specified at: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.3/guide/service/annotations/#legacy-cloud-provider
The annotation service.beta.kubernetes.io/aws-load-balancer-type is used to determine which controller reconciles the service. If the annotation value is nlb-ip or external, legacy cloud provider ignores the service resource (provided it has the correct patch) so that the AWS Load Balancer controller can take over. For all other values of the annotation, the legacy cloud provider will handle the service. Note that this annotation should be specified during service creation and not edited later.