TPS calculation for POD in Kubernetes with Prometheus

5/13/2020

I am new to kubernetes environment, i need to calculate POD based TPS using prometheus. Can any one guide me in this regard.

-- Shahid Farooq
kubernetes
prometheus

1 Answer

5/13/2020

Assuming that the http_requests_total time series all have the labels job (fanout by job name) and instance (fanout by instance of the job), you might want to sum over the rate of all instances, so you get fewer output time series, but still preserve the job dimension:

sum by (job)(rate(http_requests_total{job="pod"}[5m]))

This sequence:

rate(http_requests_total{job="pod"}[5m])

returns only the per-second rate of HTTP requests as measured over the last 5 minutes, per time series in the range vector.

rate should only be used with counters. It is best suited for alerting, and for graphing of slow-moving counters.

Note that when combining rate() with an aggregation operator (e.g. sum()) or a function aggregating over time (any function ending in _over_time), always take a rate() first, then aggregate. Otherwise rate() cannot detect counter resets when your target restarts.

If you will be interested in the total of HTTP requests you have seen in all applications, you could simply write:

sum(http_requests_total)

EDIT:

After digging in I have noticed that the official Prometheus documentation uses http_requests_total in some examples however this metric does not seem to exist in the current stable version as http://localhost:9090/metrics does not include it.

Take a look: http_requests_total. Name of this metrics has been changed from http_requests_total to promhttp_metric_handler_requests_total. Try to change name of this metrics in above examples and it should work.

Please take a look: prometheus-metrics-http-requests, PromQL, prometheus-way-of-counting.

-- MaggieO
Source: StackOverflow