We are already using node-exporters to pull/collect docker/container metrics within our kubernetes environment. Next we would also like to be able to collect application metrics (provided by spring-actuator in our case, eg. http://pod-name:8080/prometheus). How is this typically done, keeping in mind we cannot use Kubernetes services as fixed DNS names (they are load balanced over the pods behind the service) and the fact that pods disappear and appear all the time, changing their name and hence addressability?
Our initial naive approach was the following (but this will not be sufficient since it is load-balanced by the service):
- job_name: 'example-service'
scheme: http
metrics_path: '/prometheus'
static_configs:
- targets: ['example-service:8080']Any advice or examples would be really great.
Thanks a lot in advance,
Bart
I found an example in the meanwhile... question closed.
# Example scrape config for pods
#
# The relabeling allows the actual pod scrape endpoint to be configured via the
# following annotations:
#
# * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
# * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
# * `prometheus.io/port`: Scrape the pod on the indicated port instead of the default of `9102`.
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name