I have my fluentd-daemonset configured in kubernetes cluster to send logs to cloudwatch. I followed this tutorial and set fluentd. However in cloudwatch I can see that I see the logs by fluentd as well. How can I stop fluentd logs to be pushed to cloudwatch?
This is the config file I use,
<source>
@type tail
@id in_tail_container_logs
@label @containers
path /var/log/containers/*.log
pos_file /var/log/fluentd-containers.log.pos
tag *
read_from_head true
<parse>
@type json
time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source>
<label @containers>
<filter **>
@type kubernetes_metadata
@id filter_kube_metadata
</filter>
<filter **>
@type record_transformer
@id filter_containers_stream_transformer
<record>
stream_name ${tag_parts[3]}
</record>
</filter>
<match **>
@type cloudwatch_logs
@id out_cloudwatch_logs_containers
region "#{ENV.fetch('REGION')}"
log_group_name "/eks/#{ENV.fetch('CLUSTER_NAME')}/containers"
log_stream_name_key stream_name
remove_log_stream_name_key true
auto_create_stream true
retention_in_days "#{ENV.fetch('RETENTION_IN_DAYS')}"
<buffer>
flush_interval 5
chunk_limit_size 2m
queued_chunks_limit_size 32
retry_forever true
</buffer>
</match>
</label>
It should work as per annotation:
<match fluent.**>
@type null
</match>For example for this settings:
<match fluent.**>
@type file
path /var/log/internal/my-fluentd.log
compress gzip
</match>As per documentation, you can exclude those logs using this configuration:
exclude_path ["/var/log/internal/*.gz"]I excluded fluentd logs with exclude_path in the configuration as follows and now I don't get them.
<source>
@type tail
@id in_tail_container_logs
@label @containers
path /var/log/containers/*.log
pos_file /var/log/fluentd-containers.log.pos
exclude_path ["/var/log/containers/*fluentd*"]
tag *
read_from_head true
<parse>
@type json
time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source>