I am using elasticserach 6.8 and filebeat 6.8.0 in a Kubernetes cluster. I want filebeat to ignore certain container logs but it seems almost impossible :).
This is my autodiscover config
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
templates:
- condition:
contains:
kubernetes.namespace: bagmessage
config:
- type: docker
containers.ids:
- "${data.kubernetes.container.id}"
processors:
- drop_event:
when:
or:
- contains:
kubernetes.container.name: "filebeat"
- contains:
kubernetes.container.name: "weave-npc"
- contains:
kubernetes.container.name: "bag-fluentd-es"
- contains:
kubernetes.container.name: "logstash"
- contains:
kubernetes.container.name: "billing"
I've tried many variations of this configuration but still filebeats is processing container logs that I want it to ignore.
I'd like to know if what I want to do is possible and if so, what am I doing wrong?
Thanks
Seems to work now. I'm not sure what the issue was.
The first error I see in your config is incorrect indentation of the condition
section in the template
. Should be:
- type: kubernetes
hints.enabled: true
templates:
- condition:
contains:
kubernetes.namespace: bagmessage
Secondly, I'm not sure the kubernetes.*
is visible to the processors inside the config with type: docker
. You may try to reference docker.container.name
instead. Or alternatively you can move all your k8s-specific conditions to the condition
section under the templates
:
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
templates:
- condition:
and:
- contains.kubernetes.namespace: bagmessage
- contains.container.name: billing
config:
...
Also, make sure that "container.name" (but not "pod.name") is indeed what you want.