We are having a query regarding memory consumption on kafka scale out. It would be very helpful if you can give suggestion/solution for the below query. We are running kafka as docker container on kubernetes.

Memory limit of 4GiB is configured for Kafka broker POD. With some large load Kafka broker POD’s memory reached 4GiB. So we decided to manually scale out Kafka broker POD replicas from 1 to 3. But after scale out, for same load each Kafka broker PODs are consuming 4GiB memory. We expected Kafka broker POD’s memory consumption to ~1.33GiB as we are running 3 PODs for same amount of load.

Before Kafka Broker Scale out : 1 broker 6 topics each with 1 partition each Memory consumption: 4GiB

After Kafka Broker Scale out and rebalancing topics over all the brokers: 3 broker 6 topics each with 1 partitions each Memory consumption: 10GiB (Pod1: 2GiB, Pod2: 4GiB, Pod3: 4GiB)

After Kafka Broker Scale out and rebalancing topics over all the brokers: 3 broker 6 topics each with 3 partitions each Memory consumption: 12GiB (Pod1: 4GiB, Pod2: 4GiB, Pod3: 4GiB)

All deployments are tested with same amount of load. Replication factor for all topics is 1.

Edit 1