K
Q

How can fluent-bit add custom metadata to each event message being sent to splunk

8/12/2019

I'm using fluent-bit within Kubernetes to forward logs to Splunk. We'll be using the same Splunk index for multiple Kubernetes clusters, so I want to tag each event being forwarded from fluent-bit with the cluster that it comes from.

I tried using the modify functionality to "Add" or "Set" a new field in the event.

fluent-bit-filter.conf: |-
   [FILTER]
       Name                kubernetes
       Match               kube.*
       Kube_Tag_Prefix     kube.var.log.containers.
       Kube_URL            https://kubernetes.default.svc:443
       Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
       Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
       K8S-Logging.Parser  On
       K8S-Logging.Exclude On
       Add cluster devcluster

Sample log that I actually receive (missing the newly added field "cluster")

[305] kube.var.log.containers.calico-node-xzwnv_kube-system_calico-node-a4a6a2261a76ec419e9cf13ae39732b3e918726573cf1a0dece648e679011578.log: [1565578883.799679612, {"log"=>"2019-08-12 03:01:23.799 [INFO][68] int_dataplane.go 830: Received interface update msg=&intdataplane.ifaceUpdate{Name:"cali5d1a7318787", State:"up"}
-- truncj
fluent-bit
kubernetes
logging
splunk

Similar Questions

How to declare a long list of ports (in thousands)? (over 262144 characters)
How to connect mysql kubernetes container internally with nodejs k8s container?
How to properly run Python using Docker/Kubenertes?
Trying to copy files to Pods with `kubectl cp`, But getting Error: unknown flag: --all-namespaces
Kubernetes Cron Job to trigger a Jenkins pipeline

2 Answers

8/12/2019

Figured it out. You have to use a separate filter named modify. When using the helm chart, you need to add a section called rawConfig in the values.yaml and put your additional filter in between the @INCLUDE for fluent-bit-filter.conf

rawConfig: |-
 @INCLUDE fluent-bit-service.conf
 @INCLUDE fluent-bit-input.conf
 @INCLUDE fluent-bit-filter.conf
 [FILTER]
     Name modify
     Match *
     Add cluster devcluster
 @INCLUDE fluent-bit-output.conf
-- truncj
Source: StackOverflow
9/3/2019

To those of you who work with configmap.yaml add this section:

  filter.conf: |
      [FILTER]
          Name modify
          Match *
          Add KEY VALUE
-- Amit Baranes
Source: StackOverflow