I have a sandbox Kubernetes cluster, in which I shutdown all pods at night, so it can scale down with the cluster-autoscaler add-on.
The problem is, it almost always keep the master plus 2 nodes running.
Looking into cluster-autoscaler logs, I see the problem seems to be this:
Fast evaluation: node ip-172-16-38-51.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: dns-controller-3586597043-531v5
Fast evaluation: node ip-172-16-49-207.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: heapster-564189836-3h2ts
Fast evaluation: node ip-172-16-49-207.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: kube-dns-1321724180-c0rjr
Fast evaluation: node ip-172-16-49-207.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: kube-dns-autoscaler-265231812-dv17j
Fast evaluation: node ip-172-16-49-207.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: kubernetes-dashboard-2396447444-6bwtq
Fast evaluation: node ip-172-16-49-207.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: monitoring-influxdb-grafana-v4-50v9d
Fast evaluation: node ip-172-16-51-146.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: cluster-autoscaler-776613730-kqgk2and because those pods are spread, cluster-autoscaler ends up keeping 2 or more nodes up even when there is nothing running in the default namespace...
Is there a way of forcing or inducing Kubernetes to schedule all those pods together?
The idea is to make the cluster run at night with the master plus one node only. If there isn't, I was thinking on add a Scheduled Action to the AutoScale Group, so it would be forced to run all in the same node.
The solution could be to label a node, and then use the nodeSelector in the deployment to indicate that is has to run at that node. This is certainly not an advised solution since it will break once you scale to much.
An alternative to using nodeselector would be to use inter-pod affinity to ensure that your pods get packed better.
'Preferred' rather than 'Required' affinity can be used to ensure that Kubernetes will try and schedule your pods together on the same node, but if it cannot, they will schedule on different nodes.
From the documentation, it will let you specify rules like:
This pod should (or, in the case of anti-affinity, should not) run in an X if that X is already running one or more pods that meet rule Y. X is a topology domain like node, rack, cloud provider zone, cloud provider region, etc.