We want our Prometheus installation to scrape the metrics of both containers within a pod. One container exposes the metrics via HTTPS at port 443, whereas the other container exposes them via HTTP at port 8080. Both containers provide the metrics at the same path, namely /metrics.
If we declare the prometheus.io/scheme to be either http or https, only one container will be scraped. For the other one we always receive: server returned HTTP status 400 Bad Request The same happens if we do not define the prometheus.io/scheme at all. Prometheus will then use http for both ports, and fail for the container that exposes the metrics at port 443 as it would expect HTTPS requests only.
Is there a way to tell prometheus how exactly it shall scrape the individual containers within our deployment? What are feasible workarounds to acquire the metrics of both containers?
Kubernetes: 1.10.2
Prometheus: 2.2.1
apiVersion: apps/v1
kind: Deployment
metadata:
name: xxx
namespace: xxx
spec:
selector:
matchLabels:
app: xxx
template:
metadata:
labels:
app: xxx
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
spec:
containers:
- name: container-1
image: xxx
ports:
- containerPort: 443
- name: container-2
image: xxx
ports:
- containerPort: 8080- job_name: kubernetes-pods
scrape_interval: 1m
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
kubernetes_sd_configs:
- api_server: null
role: pod
namespaces:
names: []
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
separator: ;
regex: "true"
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
separator: ;
regex: (.+)
target_label: __metrics_path__
replacement: $1
action: replace
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
separator: ;
regex: ([^:]+)(?::\d+)?;(\d+)
target_label: __address__
replacement: $1:$2
action: replace
- separator: ;
regex: __meta_kubernetes_pod_label_(.+)
replacement: $1
action: labelmap
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: kubernetes_namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: kubernetes_pod_name
replacement: $1
action: replaceYou are not limited to adding prometheus annotations in deployment manifests. In fact i add them on kubernetes service manifest.
That means you can add 2 services one for container-1 and the other for container-2. Bith will be scraped by prometheus
I found a GIST snippet that takes the port from the container directly if it is named "metrics", instead of relying on a per-pod annotation. It also contains a comments to make this a regex for any port that starts with "metrics".
Maybe you can extend it to also extract the schema from the port name, like "metrics-http" and "metrics-https".
https://gist.github.com/bakins/5bf7d4e719f36c1c555d81134d8887eb
# Example scrape config for pods
#
# The relabeling allows the actual pod scrape endpoint to be configured via the
# following annotations:
#
# * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
# * `prometheus.io/path`: If the metrics path is not `/metrics` override this. This
# will be the same for every container in the pod that is scraped.
# * this will scrape every container in a pod with `prometheus.io/scrape` set to true and the
port is name `metrics` in the container
# * note `prometheus.io/port` is no longer honored. You must name the port(s) to scrape `metrics`
# Also, in some of the issues I read, there was mention of a container role, but I couldn't get
# that to work - or find any more info on it.
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_container_port_name]
action: keep
regex: metrics
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [ __address__, __meta_kubernetes_pod_container_port_number]
action: replace
regex: (.+):(?:\d+);(\d+)
replacement: ${1}:${2}
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name