iptables is growing gradually on Kubernetes nodes

3/11/2020

We have an in house Kubernetes 1.15 cluster which is running on bare metal (Ubuntu 18.04) and consists of 7 workers and 1 master node. We have many namespaces and each namespace is hosting 25+ different components and some of these components need to be exposed to the outside via NodePort (having a reverse proxy for these components is not an option).

The problem is I noticed a gradual increase in the iptables size. Over a 2 weeks period around 400 lines were added to the iptables without we change the components running on the cluster and there is nothing else running on those machines. Since this happened before I am saving iptables and when I check I don't see any strange thing inside the iptables, it is just the normal rules required for the cluster. A big iptable can cause many problems including network packet drop and I am looking to find a solution for this.

I noticed this post on Kubernetes blog but that is just a tracking tool and not a tool that solve the issue: https://kubernetes.io/blog/2019/04/19/introducing-kube-iptables-tailer/

-- AVarf
iptables
kubernetes

1 Answer

3/11/2020

What exactly bother you ?

  • Is it the size of the iptables ?
  • Do you have performance issue when scheduling a new service (meaning it takes a lot of time for your service to obtain an IP) ?

If it's just the size, then that is to be expected when you start to have undreds of pods/services. This is just how iptable works.

If it's a performance issue, then that means that you probably have thousands of components. If that's the case, Kubernetes 1.11 introduced IPVS to solve the issue.

Anyway, I invite you to have a look at those two articles :

  1. https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/
  2. https://www.projectcalico.org/comparing-kube-proxy-modes-iptables-or-ipvs/

You should have a better understanding as to why you have a lot of entries in your iptable or why it is slow after reading them.

-- Marc ABOUCHACRA
Source: StackOverflow