I need Prometheus to scrape several mongodb exporters one after another in order to compute a valid replication lag. However, the targets are scraped with a difference of several dozen seconds between them, which makes replication lag impossible to compute.
The job yaml is below:
- job_name: mongo-storage
honor_timestamps: true
scrape_interval: 1m
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
static_configs:
- targets:
- mongo-1a-exporter.monitor:9216
- mongo-2a-exporter.monitor:9216
- mongo-3a-exporter.monitor:9216
- mongos-exporter.monitor:9216
- mongo-1b-exporter.monitor:9216
- mongo-2b-exporter.monitor:9216
- mongo-3b-exporter.monitor:9216
labels:
cluster: mongo-storage
This isn't possible, Prometheus makes no guarantees about the phase of scrapes or rule evaluations. Nor is this something you should depend upon, as it'd be very fragile.
I'd aim for knowing the lag within a scrape interval, rather than trying to get it perfect. You generally care if replication is completely broken, rather than if it's slightly delayed. A heartbeat job could also help.
This isn't possible with Prometheus... normally.
However it might be possible to exploit the prometheus/pushgateway to achieve what you want. My thinking is that you write a script/tool to scrape the mongo exporters in a synchronised way, threads/forks/whatever, and then push those metrics into a prometheus/pushgateway instance.
Then configure prometheus to scrape the prometheus/pushgateway instead of the mongo exporters, and since all the metrics are in the one endpoint they will hopefully always be in sync and avoid any lag regarding being up to date.
Hope this helps.