Prometheus with multi-container pod on kubernetes

3/30/2017

I have a multi-container pod in my kubernetes deployment:

  • java
  • redis
  • nginx

For every of those containers, there's a container with Prometheus exporter as well.

The question is how can I expose those ports to Prometheus if annotations section supports only one port per pod?

annotations:
  prometheus.io/scrape: 'true'
  prometheus.io/port: 'xxxx'

but I need something like this:

annotations:
  prometheus.io/scrape: 'true'
  prometheus.io/port_1: 'xxxx'
  prometheus.io/port_2: 'yyyy'
  prometheus.io/port_3: 'zzzz'

Maybe there's some other method to scrape all metrics from my multi-container pods? Thanks in advance for any kind of help.

-- cardinal-gray
docker
kubernetes
monitoring
prometheus

2 Answers

3/31/2017

The annotations you propose should work. Create one scrape_config per port annotation, keeping only targets matching the corresponding annotation port name.

-- brian-brazil
Source: StackOverflow

10/19/2017

Here's an example job for Prometheus. Put it in your own config.

Next, add:

annotations:
   prometheus.io/scrape: 'true'

to your pod metadata.

And on every container, which provides /metrics to prom, create an appropriate port, named metrics.

That's it. Prometheus would scrape only those ports, and there would be no situation, like when your redis instance would get http requests on its 6379 port.

-- cardinal-gray
Source: StackOverflow