EFK Kubernetes Log Aggregation

9/26/2019

I am installing EFK stack to 100 days old cluster. Fluentd will pick up the logs of 100 days and will start sending to Elastic. Is there a provision that fluentd starts aggregating the log from today and not from beginning life cycle of the cluster?

-- Arpan Sharma
efk
kubernetes

1 Answer

9/30/2019

If you look into Fluentd documentation, you can find limit_recently_modified flag, which allows limit the watching files that the modification time is within the specified time range.

Here's how the limit_recently_modified can be used in conf file:

...
<source>
    exclude_path ["/var/log/wedge/*/*/MattDaemon*.log"]
    path_key source
    format none
    read_from_head true
    tag foo.*
    path /var/log/wedge/*/*/*.log
    pos_file /var/log/td-agent/wedgelog
    limit_recently_modified 86400s
    @type tail
</source>
...

Another option is to use Filebeat (instead of Fluentd), where you can find ignore_older flag. Filebeat ignores any files that were modified before the specified timespan.

I hope it will helps you.

-- muscat
Source: StackOverflow