Changing tags dynamically

1/14/2018

My goal is to be able to change log tags based on their content, and then dump them to a special, non-default index in elastic (the default index name is logstash-%date%).
I use the https://github.com/uken/fluent-plugin-elasticsearch plugin in order to be able to dump logs to elastic (with the following default configmap).

However, I will try to summarize the relevant parts of the config so a person which is not familiar with kubernetes but also with vanilla fluentd will be able to understand.

Here is the config file:

<source>
  tags all pods logs with "kubernetes.*" tag
</source>

<filter kubernetes.**>
  @type grep
  <regex>
    key name
    pattern ^bi$
  </regex>
</filter>

<filter kubernetes.**>
  @type record_transformer
  <record>
    tag bi
  </record>
</filter>

<match bi>
  @type elasticsearch
  host ...
  port ...
  index_name bi
</match>

What I'm trying to do here, is first filter all logs tagged with "kubernetes.**", and check if their "name" field contains "bi" (throwed from my app).
Then, the next filter should change the tag to "bi" for logs that matched the previous filter.
Finally, I try to catch the logs tagged with "bi" and dump them to a special index named "bi".

However, the configuration doesn't manage to change the log tags.

-- Jjang
elasticsearch
fluentd
kubernetes
logging

0 Answers