I am using prometheus-operator to manage a Prometheus deployment on my Kubernetes cluster. The setup is working fine to extract metrics from a number of my application pods, using several ServiceMonitors which select the Kubernetes endpoints giving the network address at which metrics are published. As seems to be typical (required?) with prometheus-operator, the Prometheus configuration is generated for me based on the Kubernetes endpoints discovered based on the ServiceMonitors.
I would also like my Prometheus deployment to retrieve the cAdvisor metrics published by kubelet on each cluster node. I've verified that kubelet on my cluster has cAdvisor and that it is enabled (by visiting port 4194 and observing the native cAdvisor web interface). However, what I'm missing is how to tell prometheus-operator to configure my Prometheus deployment with targets including each of these kubelet/cAdvisor servers.
The only "documentation" I've found on this is a prometheus-operator github issue asking why some cAdvisor metrics are being discovered on the poster's cluster. The explanation suggests that Kubernetes endpoints for kubelet/cAdvisor gets created by prometheus-operator somehow and then an additional ServiceMonitor finds them and causes Prometheus to be configured with additional targets. However, these Kubernetes endpoints do not exist on my Kubernetes cluster and I haven't found any information about why they ever would.
What do I need to configure so that my prometheus-operator-configured Prometheus deployment can get these metrics?
There turned out to be two problems preventing the collection of the cAdvisor metrics.
First, there is an option in prometheus-operator that must be enabled to turn on a feature of the operator which creates and maintains a kubelet service and endpoints (since kubelet does not have these normally). After adding --kubelet-service=kube-system/kubelet --config-reloader-image=quay.io/coreos/configmap-reload:v0.0.1 to my operator configuration, the desired kubelet endpoints appeared (I'm not sure what the second option does or if it's necessary; both are just copied from the linked docs).
Next, the ServiceMonitor has to be selected by the Prometheus configuration. The ServiceMonitor from the prometheus-operator docs that matches the kubelet endpoints has some labels but nothing that's guaranteed to match an already-existing Prometheus resource definition. After updating the ServiceMonitor's labels so they cause it to be selected by the existing Prometheus, the cAdvisor stats quickly become available to the Prometheus deployment.