We use EFK-stack, where F stands for fluent bit. In my kotlin spring boot application I configured logging as follows with logback and logstash
<appender name="STDOUT_JSON" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder" >
<timestampPattern>yyyy-MM-dd' 'HH:mm:ss.SSS</timestampPattern>
<fieldNames>
<timestamp>timestamp</timestamp>
<logger>logger</logger>
<version>[ignore]</version>
</fieldNames>
</encoder>
</appender>
We run that application in kubernetes. Now, sometimes for very verbose exceptions i.e. folowing, we see in kibana not parsed log entry. So there is neither logger
detected by kibana, nor message
, although those fields are in json.
{"timestamp":"2018-07-11 12:59:40.973","message":"Container exception","logger":"org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer","thread_name":"org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1","level":"ERROR","level_value":40000,"stack_trace":"org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition offer-mail-crawler-new-mails-2 at offset 2. If needed, please seek past the record to continue consumption.
Caused by: org.apache.kafka.common.errors.SerializationException: Can't deserialize data [[123, 34, 101, 118, 101, 110, 116, 73, 100, 34, 58, 34, 98, 51, 57, 100, 49, 102, 54, 49, 45, 99, 57, 51, 53, 45, 52, 48, 52, 53, 45, 57, 52, 51, 51, 45, 98, 49, 100, 98, 98, 54, 97, 57, 49, 48, 49, 53, 34, 44, 34, 101, 118, 101, 110, 116, 84, 105, 109, 101, 34, 58, 123, 34, 110, 97, 110, 111, 34, 58, 50, 56, 51, 49, 50, 57, 48, 48, 48, 44, 34, 101, 112, 111, 99, 104, 83, 101, 99, 111, 110, 100, 34, 58, 49, 53, 51, 49, 51, 49, 50, 48, 56, 48, 125, 44, 34, 101, 118, 101, 110, 116, 86, 101, 114, 115, 105, 111, 110, 34, 58, 34, 50, 48, 49, 56, 45, 48, 55, 45, 49, 49, 34, 44, 34, 115, 104, 97, 114, 101, 100, 77, 97, 105, 108, 98, 111, 120, 34, 58, 34, 111, 102, 102, 101, 114, 115, 46, 116, 101, 115, 116, 64, 97, 107, 101, 108, 105, 117, 115, 46, 100, 101, 34, 44, 34, 97, 122, 117, 114, 101, 83, 116, 111, 114, 97, 103, 101, 77, 97, 105, 108, 65, 115, 69, 109, 108, 66, 108, 111, 98, 78, 97, 109, 101, 34, 58, 34, 55, 97, 98, 54, 49, 57, 52, 97, 45, 99, 57, 101, 98, 45, 52, 55, 99, 53, 45, 56, 53, 54, 51, 45, 56, 52, 54, 54, 53, 48, 99, 51, 52, 57, 99, 48, 47, 109, 105, 109, 101, 45, 99, 111, 110, 116, 101, 110, 116, 46, 101, 109, 108, 34, 44, 34, 97, 122, 117, 114, 101, 83, 116, 111, 114, 97, 103, 101, 65, 116, 116, 97, 99, 104, 109, 101, 110, 116, 66, 108, 111, 98, 78, 97, 109, 101, 115, 34, 58, 91, 93, 44, 34, 102, 114, 111, 109, 34, 58, 34, 82, 111, 109, 97, 110, 46, 84, 117, 99, 104, 105, 110, 64, 97, 107, 101, 108, 105, 117, 115, 46, 100, 101, 34, 44, 34, 115, 117, 98, 106, 101, 99, 116, 34, 58, 34, 116, 101, 115, 116, 34, 125]] from topic [new-mails]
Caused by: com.fasterxml.jackson.module.kotlin.MissingKotlinParameterException: Instantiation of [simple type, class com.akelius.crawledmails.NewMailEvent] value failed for JSON property azureStorageMailUuid due to missing (therefore NULL) value for creator parameter azureStorageMailUuid which is a non-nullable type
at [Source: [B@66872193; line: 1, column: 350] (through reference chain: com.akelius.crawledmails.NewMailEvent["azureStorageMailUuid"])
at com.fasterxml.jackson.module.kotlin.KotlinValueInstantiator.createFromObjectWith(KotlinValueInstantiator.kt:53)
at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:138)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:471)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1191)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:314)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)
at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1626)
at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1237)
at org.springframework.kafka.support.serializer.JsonDeserializer.deserialize(JsonDeserializer.java:86)
at org.apache.kafka.common.serialization.ExtendedDeserializer$Wrapper.deserialize(ExtendedDeserializer.java:65)
at org.apache.kafka.common.serialization.ExtendedDeserializer$Wrapper.deserialize(ExtendedDeserializer.java:55)
at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:918)
at org.apache.kafka.clients.consumer.internals.Fetcher.access$2600(Fetcher.java:93)
at org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.fetchRecords(Fetcher.java:1095)
at org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.access$1200(Fetcher.java:944)
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:567)
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:528)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1086)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:628)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
"}