Kubernetes Prometheus exec inside pod and swtich to directory

1/4/2022

I want to exec inside pod and switch to directroy /opt/monitoring that contains example.json file, execute command cat example.json that will retrieve data from the file and expose that as additional target which can be discovered by Prometheus.

Is this possible to achive ?

-- Igi1
kubernetes
prometheus

1 Answer

1/4/2022

Prometheus itself can read a properly-shaped JSON file with file_sd_config:

- job: foo
  file_sd_configs:
  - files:
    - /opt/monitoring/example.json

The format of the JSON file should be like this:

[
  { "targets": [ "<host>", ... ], 
    "labels": { "<labelname>": "<labelvalue>", ... }
  }, 
  ... 
]

If you need to generate the file, you can modify container startup command so that it will create the file prior starting Prometheus:

containers:
- name: prometheus
  image: prom/prometheus
  command: ["/bin/sh", "-c"]
  args: 
  - echo '[{"targets":["example.com"],"labels":{"foo": "bar"}}]' > /tmp/filesd.json && prometheus --arg1 --arg2

Alternatively, if you need to repeat the command to generate the file, consider using cron (you have to add it to the image and start as a service before Prometheus).

At last, you can make a simple app that will expose the same JSON over HTTP and run the app alongside Prometheus (in the same pod). In such case you will need http_sd_config.

-- anemyte
Source: StackOverflow