I've been looking for specific information about high availability deployments of Kafka Connect connectors but found nothing.
In my case I have a Mongodb source connector deployed using the Confluent Helm chart. This chart supports setting the number of replicas.
Is setting replicaCount
to a value >1 enough or there are other factors to consider (tasks.max
, ...)?
If you want highly available workers, then it's pod replicas, yes.
If you want distributed tasks across workers, that's tasks.max
; if one worker dies, then tasks get rebalanced