I am installing EFK stack to 100 days old cluster. Fluentd will pick up the logs of 100 days and will start sending to Elastic. Is there a provision that fluentd starts aggregating the log from today and not from beginning life cycle of the cluster?
If you look into Fluentd documentation, you can find limit_recently_modified
flag, which allows limit the watching files that the modification time is within the specified time range.
Here's how the limit_recently_modified
can be used in conf file:
...
<source>
exclude_path ["/var/log/wedge/*/*/MattDaemon*.log"]
path_key source
format none
read_from_head true
tag foo.*
path /var/log/wedge/*/*/*.log
pos_file /var/log/td-agent/wedgelog
limit_recently_modified 86400s
@type tail
</source>
...
Another option is to use Filebeat (instead of Fluentd), where you can find ignore_older
flag. Filebeat ignores any files that were modified before the specified timespan.
I hope it will helps you.