How do I configure a kubernetes cluster to use flat networking

9/26/2019

I want to setup a kubernetes cluster in KVM VMs that uses a flat DHCP network for all its communications.

The VNICs of the VMs, which are going to be kube-master and kube-workers are bridged to use an existing physical DHCP network.

I want to know if it is possible to make kubernetes to use the same physical DHCP network both for PODs and kubernetes internalities. Does anybody know how to configure it?

I tried to configure in /etc/cni/net.d a brige networking, but it did not work. Do I have to try IPVLAN, or anything else?

Thanks

-- jfranzoy
kubernetes
networking

1 Answer

9/26/2019

I recommend reading Kubernetes documentation about Cluster Networking.

You might use Multus (a Multi Network plugin). The CNI enables attaching multiple network interfaces to pods in Kubernetes.

Multus is a Multi CNI plugin to support the Multi Networking feature in Kubernetes using CRD based network objects in Kubernetes.

Multus supports all reference plugins (eg. Flannel, DHCP, Macvlan) that implement the CNI specification and 3rd party plugins (eg. Calico, Weave, Cilium, Contiv). In addition to it, Multus supports SRIOV, DPDK, OVS-DPDK & VPP workloads in Kubernetes with both cloud native and NFV based applications in Kubernetes.

With it you can use dhcp plugin

With dhcp plugin the containers can get an IP allocated by a DHCP server already running on your network. This can be especially useful with plugin types such as macvlan. Because a DHCP lease must be periodically renewed for the duration of container lifetime, a separate daemon is required to be running. The same plugin binary can also be run in the daemon mode.

There is also NPF DHCPd on Kubernetes

This deployment uses several kubernetes features, these are mostly driven by assumptions and limitations of the old isc-dhcpd. The key points are:

  • Isc-dhcpd must know the address of itself and its failover peers at config time, this obviously conflicts with kubernetes pod addressing and self-healing features, isc-dhcpd does however allow DNS resolution of these values, so we use kubernetes dns records for pods and services to overcome these
  • Isc-dhcpd stores its dhcp leases in a file, but it does not create the file automatically on startup. To ensure this always works as intended we use an init container, to touch the file before starting dhcpd itself
  • While a pod can DNS query its own FQDN to find its own IP, it cannot query other pods IPs by their FQDN, this derailed the process for a while, but it's actually neatly solved by making a headless service, which will expose its pods IP addresses via DNS.
  • The deployment and services are actually split into two; primary and secondary. This is a bit of a workaround, because dhcpd requires the configs for each instance to be slightly different, so we accomplish this by having two of everything, the advantage of still doing a deployment is that we get a replication controller to spin up a replacement instance if one fails.
  • The two deployments share most of their config, except from a few details, so we can have a kubernetes configmap with the shared config, and then the few different details which kubernetes will mount inside the pod, on the same location for the primary and secondary.
  • Since we have two deployments, we want to make sure they don't get deployed on the same physical node in the cluster, for this we use anti-affinity so kubernetes will prefer to place it on separate nodes.
  • It would possibly be more clean to use the new statefulset, so we could avoid the duplicate deployments, this is something to look into for NPF 2018. We do not have support for PersistentVolumeClaims, but only PersistentVolumes in our iSCSI SAN, which is maybe a bit hard to do with statefulsets atm.

Update

I think this might be the best solution for you, Simple bridge network for kubernetes. You have the whole deployment in bridget.yaml which is creating a DeamonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) so it will run on all (or some) nodes will run a copy of the pod.

bridget - it's short shell script, that helps you for organise simple bridge network for Kubernetes. There is no overlays, no politics. Just flat L2-network across all your hosts and pods.

In addition bridget can automatically configure VLAN and bridge interfaces for that. See the picture:

bridget automatically retrieves node cidr from your pod-network, and configures cni for use it.

-- Crou
Source: StackOverflow