I need a suggestion on managing thousands of services and statefulsets(1 pod) in a Kubernetes cluster.
Each of the pods needs at least 500mb of memory and these statefulsets are not always up, it will be down for some time & run for some time.
What kind of nodes I should use and what kind of tools I should use here to reduce the billing?
Thank you
The type of nodes you need to use would be based on the type of workload your pods would be performing. https://cloud.google.com/compute/docs/machine-types
You reduce the billing of the cluster by always maximizing resource utilization.
Cluster Autoscaller will help you achieve the same. Increases /Decreases the number of nodes in the cluster based on load. https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler
You can also reduce your billing by using Preemptible VM's or Committed Use Discounts
Preemptible VM's Since all the nodes in the Kubernetes cluster can be replaced by a similar VM. (cattle not pets!) https://cloud.google.com/compute/docs/instances/preemptible
Committed Use Discounts GCP also reduces the cost of the VM's if you are committed to using the VM's for a longer duration https://cloud.google.com/compute/docs/instances/signing-up-committed-use-discounts
You can use https://cloud.google.com/kubernetes-engine/docs/how-to/preemptible-vms, but please note that your application should be designed to handle potential outages caused by rotation of preemptible nodes i.e. application should be fault-tolerant.
Also you can use cluster autoscaler https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler so your cluster size will adapt to the actual demand.