Recently my kafka producer running as a cronjob on a kubernetes cluster has started doing the following when pushing new messages to the queue:
{"@version":1,"source_host":"<job pod name>","message":"[Producer clientId=producer-1] Resetting the last seen epoch of partition <topic name> to 4 since the associated topicId changed from null to JkTOJi-OSzavDEomRvAIOQ","thread_name":"kafka-producer-network-thread | producer-1","@timestamp":"2022-02-11T08:45:40.212+0000","level":"INFO","logger_name":"org.apache.kafka.clients.Metadata"}
This results in the producer running into a timeout:
"exception":{"exception_class":"java.util.concurrent.ExecutionException","exception_message":"org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for <topic name>:120000 ms has passed since batch creation", stacktrace....}
The logs of the kafka consumer pod and kafka cluster pods don't show any out-of-the-ordinary changes.
Has anyone seen this behavior before and if yes, how do I prevent it?
Reason:
the Java API mode generator cannot connect with Kafka
Solution:
On each server Add a sentence to the properties file
host.name = server IP;