My goal is to be able to change log tags based on their content, and then dump them to a special, non-default index in elastic (the default index name is logstash-%date%
).
I use the https://github.com/uken/fluent-plugin-elasticsearch
plugin in order to be able to dump logs to elastic (with the following default configmap).
However, I will try to summarize the relevant parts of the config so a person which is not familiar with kubernetes but also with vanilla fluentd will be able to understand.
Here is the config file:
<source>
tags all pods logs with "kubernetes.*" tag
</source>
<filter kubernetes.**>
@type grep
<regex>
key name
pattern ^bi$
</regex>
</filter>
<filter kubernetes.**>
@type record_transformer
<record>
tag bi
</record>
</filter>
<match bi>
@type elasticsearch
host ...
port ...
index_name bi
</match>
What I'm trying to do here, is first filter all logs tagged with "kubernetes.**", and check if their "name" field contains "bi" (throwed from my app).
Then, the next filter should change the tag to "bi" for logs that matched the previous filter.
Finally, I try to catch the logs tagged with "bi" and dump them to a special index named "bi".
However, the configuration doesn't manage to change the log tags.