High availability configuration for Kafka Connect Mongodb source connector

12/31/2020

I've been looking for specific information about high availability deployments of Kafka Connect connectors but found nothing.

In my case I have a Mongodb source connector deployed using the Confluent Helm chart. This chart supports setting the number of replicas.

Is setting replicaCount to a value >1 enough or there are other factors to consider (tasks.max, ...)?

-- codependent
apache-kafka
apache-kafka-connect
confluent-platform
kubernetes
mongodb-kafka-connector

1 Answer

12/31/2020

If you want highly available workers, then it's pod replicas, yes.

If you want distributed tasks across workers, that's tasks.max; if one worker dies, then tasks get rebalanced

-- OneCricketeer
Source: StackOverflow