Fluentd Matching tags

5/19/2018

I'm trying to figure out how can a rename a field (or create a new field with the same value ) with Fluentd

Like:

agent: Chrome ....

To:

agent: Chrome
user-agent: Chrome

but for a specific type of logs, like **nginx**.

I'm trying to user record_reformer but it doesn't apply for a second filter:

<filter kubernetes.**.nginx-ingress-controller-**.log>
  @type parser
  format /^(?<host>[^ ]*) (?<domain>[^ ]*) \[(?<x_forwarded_for>[^\]]*)\] (?<server_port>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+[^\"])(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")? (?<request_length>[^ ]*) (?<request_time>[^ ]*) (?:\[(?<proxy_upstream_name>[^\]]*)\] )?(?<addr>[^ ]*) (?<response_length>[^ ]*) (?<response_time>[^ ]*) (?<status>[^ ]*)$/
  time_format %d/%b/%Y:%H:%M:%S %z
  key_name log
  types server_port:integer,code:integer,size:integer,request_length:integer,request_time:float,upstream_response_length:integer,upstream_response_time:float,upstream_status:integer
  reserve_data true
</filter>

<filter kubernetes.**>
    @type kubernetes_metadata
</filter>

<filter kubernetes.**>
 @type grep
 <regexp>
     key $.kubernetes.labels.fluentd
     pattern true
 </regexp>
</filter>

<filter kubernetes.**.deployment-name**>
  @type record_transformer
  <record>
    level ${record["Level"]}
  </record>
</filter>

<match kubernetes.**>
  @type elasticsearch
  include_tag_key true
  host "#{ENV['OUTPUT_HOST']}"
  port "#{ENV['OUTPUT_PORT']}"
  scheme "#{ENV['OUTPUT_SCHEME']}"
  reload_connections true
  logstash_format true
</match>

<match kubernetes.**>
  @type record_reformer
  remove_keys log,kubernetes
  tag mytag.generic
  <record>
    name ${record['kubernetes']['labels']['app']}
    namespace ${record['kubernetes']['namespace_name']}
  </record>
</match>

<match kubernetes.**api**>
  @type record_reformer
  remove_keys log,kubernetes
  tag mytag.api
  <record>
    user_agent ${record['req']['headers']['user-agent']}
  </record>
</match>

In such a case it doesn't send the logs for the mytag.generic

-- Idir Ouhab Meskine
fluentd
kubernetes
logging

1 Answer

5/21/2018

I have a very similar use case, and like @embik said, using record_transformer seems like a more fit choice. In my case I'm trying to downcase a key in a json record fluentd is going through, here's the conf -

<filter kubernetes.**.deployment-name**>
  @type record_transformer
  <record>
    level ${record["Level"]}
  </record>
</filter>

Hope that helps.

-- Yaron Idan
Source: StackOverflow