Load balance Kafka consumer multiple instances

9/18/2019

I have a consumer that reads and writes messages to a time-series database. We have multiple instances of the time series database running as a cluster on multiple physical machines.

Our plan is to deploy the consumer on Kubernetes so I can scale if I need more instance with load-balance they all point to a single time series service that is running.

Now I getting an Issue where it's come to my mind that if I have 5 instances which consume the same topic then they work individually means( they all get message payload and save like any one instance is doing )

What we want is

we want if one consumer is busy then it will go to the next free instance but not be subscribed to by all instance running. To scale or load-balance means I want like normal load-balancing application or how spring-boot app works when you scale on Kubernetes

so is there any way to make it like a load-balancing consumer and processing only one, even consume by 1st or 2nd or 3rd like normal app work as loadbanlacer?

if anyone has ideas about this, how it going to behave and what kind of output we are going to get if doing this with Kafka Spring boot application?

-- harkesh kumar
apache-kafka
kafka-consumer-api
kubernetes
load-balancing
spring-boot

0 Answers