Which service is doing load balancing between kubernetes nodes?

6/8/2018

What is the entry point in k8s cluster? How is request routed from entry point to certain container?

-- Dmitry Nalyvaiko
kubernetes

1 Answer

6/8/2018

This question is very general, you have many types of load balancers (internal, 3rd party, Ingresses ...)

But the best answer is Kubernetes services since all of the above relays on them.

A service in kubernetes is a set of Linux iptables (or IPVS) rules that will execute a Destination Network Address Translation (DNAT) on the packets going to specific Ip addresses. In short:

1- A service will have a virtual IP address called ServiceIP or ClusterIP.

2- The user communicates with kubernetes pods (single container or set of related containers) using the clusterIP.

3- The Iptables in the node will forward the packets with the destination ClusteIP to the IP address of the associated Pod using a CNI.

The load balancing is done through the iptables, where each service will have an Iptables rule:

 # iptables -t nat -L KUBE-SERVICES 
Chain KUBE-SERVICES (2 references)
target     prot opt source               destination  
KUBE-MARK-MASQ  tcp  -- !10.244.0.0/16        10.104.192.249       /* default/hypriot: cluster IP */ tcp dpt:http
KUBE-SVC-IKNY2FZN6EXMQQCV  tcp  --  anywhere             10.104.192.249       /* default/hypriot: cluster IP */ tcp dpt:http

# kubectl get svc hypriot
NAME      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
hypriot   ClusterIP   10.104.192.249   <none>        80/TCP    11d

For this example, the deployment hypriot have a ClusterIP 10.104.192.249, the second iptables rule will forward all the packets with this destination Ip toward the CHAIN KUBE-SVC-IKNY2FZN6EXMQQCV.

To see what this CHAIN will do:

# iptables -t nat -L KUBE-SVC-IKNY2FZN6EXMQQCV
Chain KUBE-SVC-IKNY2FZN6EXMQQCV (1 references)
target     prot opt source               destination         
KUBE-SEP-JEK5XLX6ULDDGJAZ  all  --  anywhere             anywhere             /* default/hypriot: */ statistic mode random probability 0.33332999982
KUBE-SEP-WTXTLPWDUQWUHKOF  all  --  anywhere             anywhere             /* default/hypriot: */ statistic mode random probability 0.50000000000
KUBE-SEP-OQ7KPRR3BI2AFITK  all  --  anywhere             anywhere             /* default/hypriot: */

Each of the KUBE-SEP is a Service EndPoint which represents the address of a pod, where for this deployment hypriot has 3 replicas.

# kubectl get endpoints hypriot
NAME      ENDPOINTS                                       AGE
hypriot   10.244.1.14:80,10.244.2.21:80,10.244.3.153:80   11d
# kubectl get po -o wide 
NAME                       READY     STATUS    RESTARTS   AGE       IP             NODE
hypriot-587768b4f5-9dq2k   1/1       Running   0          11d       10.244.2.21    node03
hypriot-587768b4f5-czd86   1/1       Running   0          11d       10.244.3.153   node04
hypriot-587768b4f5-j22sh   1/1       Running   0          11d       10.244.1.14    node02

One of these Endpoints will be chosen and the packet will be forwarded to the associated KUBE-SEP CHAIN:

# iptables -t nat -L KUBE-SEP-JEK5XLX6ULDDGJAZ
Chain KUBE-SEP-JEK5XLX6ULDDGJAZ (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  10.244.1.14          anywhere             /* default/hypriot: */
DNAT       tcp  --  anywhere             anywhere             /* default/hypriot: */ tcp to:10.244.1.14:80

This is the last piece of the puzzle where the DNAT will occur and the new destination will be the selected pod IP (10.244.1.14 for the pod hypriot-587768b4f5-j22sh in this example), when another service Endpoint is selected, the packet will be DNATed to another pod.

you can use Iptables -v flag to check the rules that are used which will help you in the understanding process.

Some good reads: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/ https://kubernetes.io/docs/concepts/cluster-administration/networking/ https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-iptables

-- elia
Source: StackOverflow