I am working on a building a distributed system where I need to refresh a particular configuration to all the pods of two services, service-A and service-B. This configuration is handled by another service-C. A user can send an api request to service-C which needs to be propagated to all the pods of service-A and service-B.

1. Service A and Service B works on grpc and are not exposed to the world.
2. These services can be running on different nodes (so we cannot have a file system shared between these).
3. The configuration will be around 100KB on average and there will be around 100K such configurations.

Possible ideas:

1. I can create a message bus using kafka and push these configuration on that bus which is then listened by different pods. Problem with this approach is service A and service B will have 100s of pods running so there will be like 100s of consumer groups, also if a new pod comes up it has to consume the queue from the beginning which will be time consuming.

2. Using a light weight transient key value store like consul or etcd, so Service-C will push the data in consul which can then be read by Service-A and Service-B. The issue with this approach, so many pods listening to consul can cause latency.

Can someone please help me with some ideas how i can achieve this on kubernetes natively?

Thanks.