We have run 100 services in our kubernetes cluster, so the number of iptables rules grown to 900+(our services are node-port type). Then we found that the speed of creating kube-proxy has been slown down.
After analyzing the log, we found the speed of writing iptables rules is really low----it took about 50ms to write one rule, so it took about 40s to flush all iptables rules.
So I was wondering about what is the proper speed of writing iptables rules? Is my case reasonable or not?
This is a known issue. The newer kube-proxy uses iptables-save and iptables-restore to do this, which is constant-time rather than linear. We're looking at backporting that logic to the "userspace" kube-proxy version, but it probably won't make it for v1.2 release.
TL;DR: newer kube-proxy makes this a non-issue.