Auto increase Kafka partitions based on load on kubernetes

2/21/2022

We have installed Kafka on kubernetes , we want Kafka broker partitions needs to auto increase based on load from input source , so I want to know is there any possibile way to increase partitions based on load on kubernete

-- veerareddy
apache-kafka
kubernetes
load

1 Answer

2/21/2022

Brokers don't have partitions that scale; topics do.

You'll have to build your own scripts for this because there are some caveats

  1. Adding partitions to topics doesn't move data, and will not reduce producer load. It might increase broker cluster load because there would now be more data be able to get written/read. The only way to reduce broker load is to add more brokers, not just partitions

    • If you add brokers, you'll need to have scripts that move partitions across brokers since that's not automatic either
  2. It'll also remove any ordering guarantee for consumers, if they were relying on it (for example, compacted topics)

-- OneCricketeer
Source: StackOverflow