I have service A which is a consumer from some queue.
I can monitor and count any consumed message, easily with Prometheus :)
from prometheus_client import start_http_server, Counter
COUNTER_IN_MSGS = Counter('msgs_consumed', 'count consumed messages')
start_http_server(8000)
while(queue not empty):
A.consume(queue)
COUNTER_IN_MSGS.inc()
But than, one day I decide to duplicate my consumer to 10 consumer which do the same {A1, A2..., A10}, by using the same code but running on 10 different dockers (containers on K8s in my case).
How can I monitor them using Prometheus?? Should I change my code and some id to each consumer as label?
What is the best practice to do in order to be able to sum them all together but also count on each by its own?
Yes, you should consider using labels to disambiguate metrics (e.g. Counters) by instance.
You'll need to determine a unique identifier to use.
Kubernetes provides a Downward API that enables you to surface information from the Pod to a container. One of these values should be useful.
You can then use PromQL ignoring to e.g. sum across Counters and ignore one or more labels.
With this approach, you get to choose whether to sum by instance or across instances.