We have a relatively standard Kubernetes cluster which is hosted in the cloud behind a load balancer. We have found that most of the system pods only run as a single instance, the really concerning thing is that by default our nginx ingress controller will also run as a single instance. This means that in the event of a node failure there is a 1/n chance of every single application going down until the liveness probe kicks in and moves the ingress controller pod.
We have had to increase the number of replicas of our ingress controller because it is a single point of failure. However, I'm not particularly happy about how that makes our network diagram look and I'd imagine that this would cause issues if any of our applications were stateful.
Some pods (like heapster) you can probably only have a single instance of but I was wondering if anyone had any guidelines on what can and can't be scaled up and why this is the default behavior?
Thanks,
Joe
The nginx ingress controller is actually a Deployment. See here.
Your namespace and deployment name may vary, but try this:
kubectl scale deployment --namespace nginx-ingress nginx-ingress-controller --replicas=3
I don't see any issues with scaling your ingress controllers, you just have more replicas and are served by your external IPs or load balancer. In the event one of them goes down your load balancer will stop forwarding requests to the ingress that is down.
As far as the backend you can have one or more replicas too, it really depends on what kind of redundancy you want to have and also the type of service. Having said that, I really don't recommend an ingress for stateful apps. An ingress is at a layer 7 (HTTP(s)), you'd better off connecting directly using TCP in your cluster. For example, connecting to a MySQL or PostgreSQL instance. I suppose ElasticSearch is one of those exceptions where you would add data through HTTP(s), but I'd be careful of posting large amounts of data through an Ingress.