I'm using fluent-bit within Kubernetes to forward logs to Splunk. We'll be using the same Splunk index for multiple Kubernetes clusters, so I want to tag each event being forwarded from fluent-bit with the cluster that it comes from.
I tried using the modify functionality to "Add" or "Set" a new field in the event.
fluent-bit-filter.conf: |-
[FILTER]
Name kubernetes
Match kube.*
Kube_Tag_Prefix kube.var.log.containers.
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
K8S-Logging.Parser On
K8S-Logging.Exclude On
Add cluster devcluster
Sample log that I actually receive (missing the newly added field "cluster")
[305] kube.var.log.containers.calico-node-xzwnv_kube-system_calico-node-a4a6a2261a76ec419e9cf13ae39732b3e918726573cf1a0dece648e679011578.log: [1565578883.799679612, {"log"=>"2019-08-12 03:01:23.799 [INFO][68] int_dataplane.go 830: Received interface update msg=&intdataplane.ifaceUpdate{Name:"cali5d1a7318787", State:"up"}
Figured it out. You have to use a separate filter named modify. When using the helm chart, you need to add a section called rawConfig in the values.yaml and put your additional filter in between the @INCLUDE for fluent-bit-filter.conf
rawConfig: |-
@INCLUDE fluent-bit-service.conf
@INCLUDE fluent-bit-input.conf
@INCLUDE fluent-bit-filter.conf
[FILTER]
Name modify
Match *
Add cluster devcluster
@INCLUDE fluent-bit-output.conf
To those of you who work with configmap.yaml add this section:
filter.conf: |
[FILTER]
Name modify
Match *
Add KEY VALUE