Kubernetes 1.18.6 ServiceTopology and Ingress Support

8/13/2020

I have set up and configured the Service Topology feature in a self-hosted Kubernetes Cluster. Everything is working as expected when I route traffic directly to a configured service using either the ClusterIP or NodePort. Topology Keys are respected, the correct node is chosen for routing to pod, etc, etc.

However, when using an ingress that is mapped to a service (tried with both the haproxy and traefik ingress), Topology aware routing is ignored. I'm not sure if this just not supported, a bug, or some configuration I am missing. The following are the methods I have tried.

1r) client curl -> Topology Keyed Service -> Deployment -> Pod

Result: Works as expected.

2r) busybox pod curl -> Topology Keyed Service -> Deployment -> Pod

Result: Works as expected.

3r) client curl -> routed clusterIP of Traefik Service Ingress -> Topology Keyed Service -> Deployment -> Pod

Result: Selected node pod is random and does not align with Topology Keys on Node.

4r) client curl -> Host Networked Haproxy Ingress DaemonSet -> Topology Keyed Service -> Deployment -> Pod

Result: Selected node pod is random and does not align with Topology Keys on Node.

From what I can tell, it seems that both the haproxy and traefik ingresses route traffic directly to the pods and bypass the service.

I've tested this by configuring the topology keys as follows (without * catch-all which should fail if not matched):

topologyKeys:
  - "kubernetes.io/hostname"
  - "topology.kubernetes.io/zone"
  - "topology.kubernetes.io/region"

When testing with the ingresses, the service will always respond, though still randomly picking a pod in the replica set. When testing with the service, it will fail when no pods are satisfied with the topology labels on the node that the inbound request was received on (as expected).

I've looked through all the docs and spent many hours already googling and looking through Github issues and can't find any mention of this as an issue or as an unsupported config, etc.

-- EDIT --

Thanks to @Rico I was able to get this working on Ingresses with both the nginx and haproxy-ingress. The fix required the following annotations on the ingress.

nginx:

annotations:
  nginx.ingress.kubernetes.io/service-upstream: "true"

haproxy-ingress:

annotations:
  ingress.kubernetes.io/service-upstream: "true"
-- Nicholas Marus
kubernetes
kubernetes-ingress

1 Answer

8/13/2020

The answer here is that typically ingress controllers talk to the Endpoint API and not the service API. It's documented on the nginx ingress controller page and the Traefik ingress controller. (Maybe not on some other ingresses)

Another alternative would be using EndpointSlices but as of today, the ingress controllers don't support that yet.

I believe the path forward here is for the ingress controllers to start supporting EndpointSlices.

Update:

The nginx ingress controller has a way to bypass the Endpoint behavior with the nginx.ingress.kubernetes.io/service-upstream annotation. "But then you cannot have custom LB algorithms or sticky sessions".

✌️

-- Rico
Source: StackOverflow