SCDF on k8s: Producer exception when sending message to Kafka - ticktock + no messages on ticktock.time topic

8/3/2017

The timer application throws a per-second error when trying to send its message to Kafka:

2017-08-03 10:20:58,805 ERROR kafka-producer-network-thread | producer-2 o.s.k.s.LoggingProducerListener:76 - Exception thrown when sending a message with key='null' and payload='{-1, 2, 11, 99, 111, 110, 116, 101, 110, 116, 84, 121, 112, 101, 0, 0, 0, 12, 34, 116, 101, 120, 116...' to topic metrics: org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for metrics-0 due to 30009 ms has passed since batch creation plus linger time

It seems to me like these would be the records for the metrics topic, but at the same time there are no records in the ticktock.timer topic in Kafka. I'm guessing this partition key being null could be the problem. But I am not so sure why it should use some partitioning setup that is not the default one.

Here's what I deployed:

dataflow:> app register --type sink --name log --uri docker://springcloudstream/log-sink-kafka-10:1.2.0.RELEASE --metadata-uri maven://org.springframework.cloud.stream.app:log-sink-kafka-10:jar:metadata:1.2.0.RELEASE dataflow:> app register --type source --name time --uri docker://springcloudstream/time-source-kafka-10:1.2.0.RELEASE --metadata-uri maven://org.springframework.cloud.stream.app:time-source-kafka-10:jar:metadata:1.2.0.RELEASE dataflow:> stream create --name ticktock --definition "time | log" --deploy

And here's my setup:

spring-cloud-dataflow-server-kubernetes: 1.3.0.BUILD-SNAPSHOT
Spring Cloud Data Flow Core: 1.3.0.BUILD-SNAPSHOT
Spring Cloud Dataflow UI: 1.3.0.BUILD-SNAPSHOT

I'm using Kafka 10 BTW.

What could be the problem?

Many thanks for your support!

-- user3350314
kubernetes
spring-cloud-dataflow

1 Answer

8/8/2017

It looks odd that the time source application's log has the producer publishing to the metrics topic. By any chance, you have this topic configured as part of this stream?

I assume the stream deployment has the topic ticktock.time created and can you share if there are any other logging info (may be some debug logs) as to what's going on with the time source application.

Also, after a quick search, it looks like the Kafka Producer batch's timeout exception does arise in some kubernetes environment and do you think you are running into one of those?

-- Ilayaperumal Gopinathan
Source: StackOverflow