How can I make my Kubernetes service name set as the Prometheus Job name automatically? I mean to say that is there a possible way to get a new service created in K8s made automatically as a target in Prometheus configuration? In Kubernetes, I will like to deploy my application as set of services.
For every service there could be more than 1 pod associated.
MApping could be done like:
But I really don't know if this is feasible with some Configuration changes in Prometheus. Please correct me if I am wrong anywhere.
If this is not possible, do I need to write create explicitly Prometheus job in the Prometheus Configuration file every time before deployment.
You will typically want metrics per pod, as you would normally have when using regular nodes instead of containers/pods.
Using this Prometheus configuration you will get a target for every pod that's running on the cluster automatically. This is the important part
# Example scrape config for pods
#
# The relabeling allows the actual pod scrape endpoint to be configured via the
# following annotations:
#
# * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
# * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
# * `prometheus.io/port`: Scrape the pod on the indicated port instead of the
# pod's declared ports (default is a port-free target if none are declared).
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_nameAs explained in the comments above, this is configured so pods that contains the prometheus.io/scrape set to true, will be scraped by Prometheus, becoming a target. Pods will then need to have a metrics endpoint exposing metrics in the Prometheus format. You can use the prometheus.io/path and prometheus.io/port to configure where Prometheus will look for the metrics on your pod.