K
Q

Headers based routing with Kubernetes

11/15/2019

Suppose we have a service replicated to several pods. The first request to the service should be randomly(or by a load balancing algorithm) routed to a pod and the mapping 'value_of_certain_header -> pod_location' should be saved somehow so next request will be routed to specific pod.

Are there any Ingress controllers or other approaches for Kubernetes to implement stickiness to specific pod by request header? Basically I need the same behaviour that haproxy does with its sticky tables.

-- Zalizniak
kubernetes
load-balancing
routing

Similar Questions

Deploying K8S cluster without default worker pool in IBM Cloud
How to remove docker images added to microk8s image cache?
Why does data read from a yaml file via set-file of helm contains extra characters?
alias for parent chart
Can kubernetes volumes be used for deployments? If so what happens if each pod is on different host?

3 Answers

11/27/2019

Kubernetes Ingress works on OSI Layer7, so it can take into account HTTP headers, but it only forwards traffic to Kubernetes Services, not to Pods.

Unfortunately, Kubernetes Services in turn, can't deliver traffic to specific pods depending on HTTP headers, because Service is basically set of iptables rules that deliver traffic to pod, only analyzing data on OSI Layer4 (IP address, tcp/udp, port number).

For example, let's look at kube-dns service iptables rules:

# kube-dns service
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU 
# random load balancing traffic between pods
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-AALXN3QQZ3U27IAI
-A KUBE-SVC-TCOU7JCQXEZGVUNU -j KUBE-SEP-WTX2W5TQOZGP42TM
# dns pod 1
-A KUBE-SEP-AALXN3QQZ3U27IAI -p udp -m udp -j DNAT --to-destination 10.244.0.16:53
# dns pod 2
-A KUBE-SEP-WTX2W5TQOZGP42TM -p udp -m udp -j DNAT --to-destination 10.244.0.17:53

I can imagine only one realistic way to deliver traffic to specific pod based on HTTP Headers.

  1. Configure custom Ingress controller which can set the headers for each Client session and use known DNS names of pods as a back-end destination points. I can't recommend particular solution, so in worst case it could be created using some examples.

  2. Kubernetes StatefulSet creates Pods with predictable names like statefulset-name-0, statefulset-name-1, etc. Corresponding Headless Service (ClusterIP: None) creates DNS names for each Pod.

For example, for StatefulSet nginx-ss with three replicas, three pods would be created and Service nginx-ss would create three DNS A records for Pods:

nginx-ss-0   1/1     Running     10.244.3.72    
nginx-ss-1   1/1     Running     10.244.3.73    
nginx-ss-2   1/1     Running     10.244.1.165   

nginx-ss-0.nginx-ss.default.svc.cluster.local. 5 IN A 10.244.3.72
nginx-ss-1.nginx-ss.default.svc.cluster.local. 5 IN A 10.244.3.73
nginx-ss-2.nginx-ss.default.svc.cluster.local. 5 IN A 10.244.1.165
-- VAS
Source: StackOverflow
11/15/2019

If you want to make sure that connections from a particular client are passed to the same Pod each time, you can select the session affinity based on client’s IP addresses by setting service.spec.sessionAffinity to “ClientIP” (the default is “None”) in the service definition YAML.

You can also set the maximum session sticky time by setting service.spec.sessionAffinityConfig.clientIP.timeoutSeconds appropriately. (the default value is 10800, which works out to be 3 hours)

-- P Ekambaram
Source: StackOverflow
11/15/2019

Assuming that 'pod_location' is inserted into HTML header by app that is running on that pod, the Ingress (and Ingress Controller) can be used to achieve header based routing.

For example, Traefik v2.0 has the new Custom Resource Definition (CRD) called IngressRoute that extends the Ingress spec and adds support for features such as Header based routing.

In the following example, I have two services: one exposing an Nginx deployment and other one exposing an Apache deployment. With the IngressRoute CRD, the match for the router will be the header X-Route:

apiVersion: traefik.containo.us/v1alpha1 
kind: IngressRoute 
metadata: 
  name: headers 
spec: 
  entrypoints: 
    - web 
    - websecure 
  routes: 
    - match: Headers(`X-ROUTE`,`Apache`) 
      kind: Rule 
      services: 
        - name: apache 
          port: 80 
    - match: Headers(`X-ROUTE`,`nginx`) 
      kind: Rule 
      services: 
        - name: nginx 
          port: 80

Full example

With the X-ROUTE: Apache header:

curl http://46.101.68.190/ -H 'X-ROUTE: Apache' 
html><body><h1>It works!</h1></body></html>

With the X-ROUTE: nginx header:

> curl http://46.101.68.190/ -H 'X-ROUTE: nginx'
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...and so on...

And Traefik provides additional info with configuration examples for their middleware .

-- Nick
Source: StackOverflow