I have added several Horizontal Pod Autoscalers (HPAs) to a Kubernetes cluster. I want to monitor the number of replicas of each pod over time.
Does StackDriver have an option to monitor the number of replicas of each pod over time? When creating a metric I can't find an option that will allow me to do this.
I don't think you can count pods, but you can count containers. With a constant number of containers inside the replicated Pods and proper filtering, you'll get what you need.
While in Stackdriver go to Dashboard -> Create dashboard. Select the following settings:
Resource type: GKE Container
Metric: Uptime
Filter: (that's actually up to you, probably some label would work)
Group by: (use this if you want to have several lines on the chart at once)
Aggregator: count
What this does is basically gather all the uptimes of containers in your cluster, filter this data through your criteria, and count how many entries are left. This gives you the number of containers that were up at given time. When a container doesn't exist, there's no entry for it, so the number of entries is lower.
If you have only one container per pod, then that's it. If you have more, just take that into account and divide the values on the chart byt the number of containers per pod.