I'm hoping you can help me, because I'm really struggling.
Our set up is that we're using fluentd to forward logs from Openshift to an external aggregator, and then on to ElasticSearch - we're using the logging solution as supplied by RedHat, but without the internal ElasticSearch/Kibana. That bit is mostly working OK, except that all the application logs are coming out tagged as 'kubernetes.journal.container' - this is the default.
I'm trying to retag them to match logs collected from our existing non-OpenShift estate and to do this I've added a label to the deploymentconfig in a sample app. I can see this label in the logs that come out, but I cannot get the retagging to work (using the rewrite_tag_filter output plugin) either at the Openshift end or at the aggregators.
We're essentially using the default configmap (as supplied with Openshift 3.7) but with a forwarder to an external service to route the route the traffic to our aggregators tacked on to the end:
<match **>
@type forward
heartbeat_type tcp
flush_interval "5s"
<server>
host external-es-host-service
port 24224
</server>
</match>
I've tried adding this match in (just above this section) to rewrite the tags - it just stops any logs being forwarded:
<match kubernetes.journal.container>
@type rewrite_tag_filter
# Retag the logs from application containers based on the kubernetes labels
rewriterule1 ${result['kubernetes']['labels']['logdetails']} ^(.+)$ $1
</match>I've also tried it with a much less restrictive regex (^(.*)$) too.
Edit: more detail: It seems that the restriction on changing the tag name at the Openshift end was due to the ViaQ plugins - among other things, they change the time field to '@timestamp'; not having a specific formatter defined was causing errors.
However, I've now trying performing the retagging at the aggregator end (running version 3.1.1 of td-agent), and I'm still getting problems. I'm running this match command:
# Match for Openshift application logs
<match kubernetes.journal.container>
@type rewrite_tag_filter
<rule>
key $['kubernetes']['labels']['logdetails']
pattern ^([^\.]+)\.([^\.]+)\.([^\.]+)\.([^\.]+)\.([^\.]+)$
tag $1.$2.$3.ocp-$4.$5.${tag}
</rule>
</match>
That works, but if I try to remove the ${tag} from the tag section, it fails (and I get no logs in Kibana). If I add a remove_tag_prefix directive, I still need to include that tag, which disappears but leaves a trailing '.'. If I remove that, the match fails (and I get no logs in kibana). There aren't any rules at the aggregator end that match the new tag (with or without ${tag}).
Can anyone help?
In OpenShift 3.6 that I was running, the version of fluentd was 0.12.39. According to these requirements, the fluent-plugin-rewrite-tag-filter version had to be <2.0.0. It appeared that the so-called "nested attributes" like ['kubernetes']['labels']['logdetails'] and described here, were not supported in fluent-plugin-rewrite-tag-filter v1.6.0.
Luckily for me, what I wanted to replace the tag with was available as top-level field (viaq_index_name). Here is how my rule looked like:
<match output_**>
@type rewrite_tag_filter
<rule>
key viaq_index_name
pattern ^(.+)$
tag $1
</rule>
</match>For whatever reason in OCP 3.6, the project names that in OCP 3.4 were used as tags were replaced with output_tag and output_ops_tag. The above rule made the project names part of the tags in OCP 3.6, which allowed me to filter log data by project.
It may be worth checking the versions of fluentd and fluent-plugin-rewrite-tag-filter and see if you can upgrade fluent-plugin-rewrite-tag-filter to a version that supports nested attributes.