On our current Rancher environment, we dynamically configure an Nginx configuration based on calls to the Rancher metadata, using labels on the containers to determine if that container is included in the Nginx routing.
We use confd with a Rancher backend to accomplish this metadata check and to dynamically change/reload a new Nginx configuration.
We have started working on migrating to Kubernetes (AWS EKS). Is there an equivalent to this confd/Rancher available for Kubernetes ?
Due to some technical reasons and time scoping reasons, we can't replace this nginx with an ingress equivalent at this time, so are looking into using annotations or labels on services/pods to keep a dynamic configuration capability.
Hosted Kubernetes providers don't generally give you direct access to the backing etcd so your best and desired option is to mine the Kubernetes API for the resources you are interested in and generate your configuration.
This is precisely what an ingress controller does - watches for changes in Kubernetes resources and generates configuration for a load balancer like nginx.
One of the nginx controllers lets you completely replace the template that it uses with one of your own.
The NGINX template is located in the file /etc/nginx/template/nginx.tmpl.
Using a Volume it is possible to use a custom template. This includes using a Configmap as source of the template
volumeMounts:
- mountPath: /etc/nginx/template
name: nginx-template-volume
readOnly: true
volumes:
- name: nginx-template-volume
configMap:
name: nginx-template
items:
- key: nginx.tmpl
path: nginx.tmplYou could deploy a version of the nginx ingress controller under a custom class like nginx-legacy so it doesn't try to expose services that are expecting the normal nginx ingress.
To do this, the option --ingress-class must be changed to a value unique for the cluster within the definition of the replication controller.
spec:
template:
spec:
containers:
- name: nginx-ingress-legacy-controller
args:
- /nginx-ingress-controller
- '--election-id=ingress-controller-leader-internal'
- '--ingress-class=nginx-legacy'
- '--configmap=ingress/nginx-ingress-internal-controller'And then annotate legacy services to assign resources to that ingress using:
metadata:
name: foo
annotations:
kubernetes.io/ingress.class: "nginx-legacy"To add a little further detail on what we eventually found thanks to Scott Anderson's answer.
Using the nginx custom template technique, we were able to dynamically configure the nginx configuration by using annotations in the Ingress resources and referencing them in the nginx custom template.
With an Ingress resource metadata defined as:
metadata:
name: foo
annotations:
kubernetes.io/ingress.class: "nginx-legacy"
mycompany/bar: "path_to_service"Within the custom Nginx template (location block), to see if the annotation is present:
{{if index $ing.Annotations "mycompany/bar"}}To get a value from an annotation:
{{$bar:= index $ing.Annotations "mycompany/bar"}}