we are using Fluentd to push kubernetes container logs to Elasticsearch. and we want to create indices on Elasticsearch in format of NAMESPACE_CONTAINERNAME. below is our config of record transformer and Elasticsearch outPut plugin.
<filter kubernetes.**>
@type record_transformer
enable_ruby true
<record>
container_name ${record["kubernetes"]["container_name"]}
namespace ${record["kubernetes"]["namespace_name"]}
pod ${record["kubernetes"]["pod_name"]}
host ${record["kubernetes"]["host"]}
app ${record["kubernetes"]["labels"]["app"]}
</record>
</filter>
Elasticsearch Output:
<label @OUTPUT>
<match **>
@type elasticsearch
host "elasticsearch-master.default.svc.cluster.local"
port 9200
logstash_format true
logstash_prefix ${namespace}_${container_name}
</match>
</label>
but when Fluentd pushing logs to Elasticsearch, indices getting created as ${namespace}_${container_name}-2021.02.25. any suggestion or guidance on this most welcome. :slightly_smiling_face:
was able to make it work by adding buffer section.
<label @OUTPUT>
<match **>
@type elasticsearch
host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
logstash_format true
logstash_prefix ${namespace}_${container_name}
<buffer tag, container_name, namespace>
@type file
path /var/log/${container_name}/app.log
</buffer>
</match>
</label>