I'm using FluentD (deployed as DaemonSet) to stream k8s app (containers) logs to elasticsearch. i want extract specific keys from the 'log' key string in elastic such as logKey: ... or statusCode:.. (inside the 'log' key below)
Please see the log i'm trying to parse:
i saw that i should use FluentD plugin to parse the log before it sent to elastic, but unfortunately i didn't find a way to do it.
this is my current ConfigMap containers section :
<source>
@type tail
path /var/log/containers/*.log
pos_file /var/log/app.log.pos
tag kubernetes.*
read_from_head true
<parse>
@type json
time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source>
Thanks a lot for your help.
Do you trying this add-on?
It seems already setup the configure of the fluentd.
https://github.com/kubernetes/kubernetes/blob/8568d1843daababe70763b30ae897388710b2216/cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml
Is your ConfigMap appended to above ConfigMap?