Fluentbit Kubernetes - How to extract fields from existing logs
I have configured EFK stack with Fluent-bit on my Kubernetes cluster. I can see the logs in Kibana.
I also have deployed nginx pod, I can see the logs of this nginx pod also in Kibana. But all the log data are sent to a single field "log" as shown below.
How can I extract each field into a separate field. There is a solution for fluentd already in this question. Kibana - How to extract fields from existing Kubernetes logs
But how can I achieve the same with fluent-bit?
I have tried the below by adding one more FILTER section under the default FILTER section for Kubernetes, but it didn't work.
[FILTER] Name parser Match kube.* Key_name log Parser nginx
From this (https://github.com/fluent/fluent-bit/issues/723), I can see there is no grok support for fluent-bit.
Similar Questions
2 Answers
Look at this configmap:
The nginx parser should be there:
[PARSER]
Name nginx
Format regex
Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
In our official documentation for Kubernetes filter we have an example about how to make your Pod suggest a parser for your data based in an annotation: