add custom fields in Elasticsearch indices while pushing logs from Fluentd

2/25/2021

we are using Fluentd to push kubernetes container logs to Elasticsearch. and we want to create indices on Elasticsearch in format of NAMESPACE_CONTAINERNAME. below is our config of record transformer and Elasticsearch outPut plugin.

      <filter kubernetes.**>
        @type record_transformer
        enable_ruby true
        <record>
          container_name ${record["kubernetes"]["container_name"]}
          namespace ${record["kubernetes"]["namespace_name"]}
          pod ${record["kubernetes"]["pod_name"]}
          host ${record["kubernetes"]["host"]}
          app ${record["kubernetes"]["labels"]["app"]}
        </record>
      </filter>

Elasticsearch Output:

    <label @OUTPUT>
      <match **>
        @type elasticsearch
        host "elasticsearch-master.default.svc.cluster.local"
        port 9200
        logstash_format true
        logstash_prefix ${namespace}_${container_name}
      </match>
    </label>

but when Fluentd pushing logs to Elasticsearch, indices getting created as ${namespace}_${container_name}-2021.02.25. any suggestion or guidance on this most welcome. :slightly_smiling_face:

-- chitender kumar
elasticsearch
fluentd
kubernetes

1 Answer

2/25/2021

was able to make it work by adding buffer section.

    <label @OUTPUT>
      <match **>
        @type elasticsearch
        host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
        port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
        logstash_format true
        logstash_prefix ${namespace}_${container_name}
        <buffer tag, container_name, namespace>
          @type file
          path /var/log/${container_name}/app.log
        </buffer>
      </match>
    </label>
-- chitender kumar
Source: StackOverflow