I'm running an elasticsearch cluster using StatefulSets on Google Container Engine (GKE) (my k8s configs are very similar to the ones here: https://github.com/pires/kubernetes-elasticsearch-cluster/tree/master/stateful)
I created the k8s cluster with --enable-stackdriver-kubernetes Now I want to also install & use the Stackdriver elasticsearch plugin: https://cloud.google.com/monitoring/agent/plugins/elasticsearch
Should I install the Stackdriver monitoring agent+plugin inside the ES pods? or on the nodes?
If you're using Stackdriver logging agent to generate/export logs for pods running elasticsearch on a Kubernetes cluster, you can have Stackdriver logging enabled for the cluster (this is enabled by default and can be enabled/disabled through the Console), the Stackdriver logging agent will be deployed on the cluster.
If the logging agent is running on the cluster, logs from each container will automatically be gathered, formatted and exported by the logging agent to Stackdriver Logging [1] for the deployed pods/containers, including elasticsearch.
Kubernetes does things differently to Compute Engine instances in terms of the monitoring agent. If Stackdriver Monitoring is enabled for the cluster, pods are deployed running the Kubernetes Engine version of the Stackdriver Agent and in the case of Kubernetes these come in the form of heapster pods as explained in more detail here [2].
[1] https://cloud.google.com/kubernetes-engine/docs/how-to/loggin
[2] https://cloud.google.com/monitoring/kubernetes-engine/customizing