Kafka Connect spamming rejoin

4/20/2020


I've finally managed to setup Kafka Connect (using Kerberos, SSL and ACL) and deployed in a Kubernetes environment. It seemed work fine until, I believe, I tried to experiment with two workers. I did this by just changing the replication-factor in my k8s-spec to 2.
However this seem to have rendered my connector in a faulty state.
Now each time i try to restart my k8s-deployment the worker seem to startup but then end up in a state spamming the log with:

[2020-04-20 13:49:02,882] INFO [Worker clientId=connect-1, groupId=orion_kafka_con] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-04-20 13:49:02,894] INFO [Worker clientId=connect-1, groupId=orion_kafka_con] Successfully joined group with generation 22981593 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2020-04-20 13:49:02,894] INFO Joined group and got assignment: Assignment{error=1, leader='connect-1-ac35d22d-73a7-4d56-a42d-7e5904a52303', leaderUrl='http://<host>:8083/', offset=228, connectorIds=[], taskIds=[]} (org.apache.kafka.connect.runtime.distributed.DistributedHerder)

The error seem to occur disregarding of how many workers I run.

From the log you can tell that there seem to be an error (error=1) but Ive been unable to find out what that points to.
I tried to resolve the issue by resetting the internal-topics (reducing retention time to 1s) but it didn't seem to make any difference.

Any suggestions on how to resolve this? I appreciate any help I can get.
Regards

edit: The logs are spewing out making (?) the REST-api unresponsive meaning I cant delete the started connector. Enabled DEBUG (with even more logs) but I cant seem to find anything wrong. No ERRORS, no WARNINGS.

edit 2: I managed to fix it but not resolve it. After resetting the kafka-connects internal topics (seems I wasnt giving the change of retention-time enough time to take affect) the problem eventually went away. I have no idea what caused the problem thou.

-- Jiinxy
apache-kafka
apache-kafka-connect
kubernetes

0 Answers