Kafka Client in kubernetes POD

4/28/2020

I am trying to identify the options to leverage Kafka consumers within kubernetes pods. Basically, I have a service which is running in kubernetes and is using an in memory data, any update needed to the data will be published to a Kafka topic by an external agent and I need a consumer to subscribe to this topic and update the in memory data in the each pod for that service(the container running in each pod for that service has an HTTP API to update the in memory data) - I am thinking of having a sidecar model kafka consumer running in each pod and update the instance of the service. Questions are

  1. Does anyone has done such a scenario already?.
  2. How to create a kafka consumer as a sidecar within a pod which is listening to a well defined topic in KAFKA?.

I have tried searching How to identify all pods for a services and push the updates directly(from a common KAFKA consumer) to the instance in each pod for that service as well - without using a kafka consumer in each pod- , But I thought having a consumer in each pod might be a better solution.

-- jereesh thomas
apache-kafka
kubernetes
pod
sidecar

1 Answer

4/28/2020

I think you're just describing a Kafka Sink Connector?

-- coderanger
Source: StackOverflow