I'm trying to figure out how can a rename a field (or create a new field with the same value ) with Fluentd
Like:
agent: Chrome ....
To:
agent: Chrome
user-agent: Chrome
but for a specific type of logs, like **nginx**
.
I'm trying to user record_reformer
but it doesn't apply for a second filter:
<filter kubernetes.**.nginx-ingress-controller-**.log>
@type parser
format /^(?<host>[^ ]*) (?<domain>[^ ]*) \[(?<x_forwarded_for>[^\]]*)\] (?<server_port>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+[^\"])(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")? (?<request_length>[^ ]*) (?<request_time>[^ ]*) (?:\[(?<proxy_upstream_name>[^\]]*)\] )?(?<addr>[^ ]*) (?<response_length>[^ ]*) (?<response_time>[^ ]*) (?<status>[^ ]*)$/
time_format %d/%b/%Y:%H:%M:%S %z
key_name log
types server_port:integer,code:integer,size:integer,request_length:integer,request_time:float,upstream_response_length:integer,upstream_response_time:float,upstream_status:integer
reserve_data true
</filter>
<filter kubernetes.**>
@type kubernetes_metadata
</filter>
<filter kubernetes.**>
@type grep
<regexp>
key $.kubernetes.labels.fluentd
pattern true
</regexp>
</filter>
<filter kubernetes.**.deployment-name**>
@type record_transformer
<record>
level ${record["Level"]}
</record>
</filter>
<match kubernetes.**>
@type elasticsearch
include_tag_key true
host "#{ENV['OUTPUT_HOST']}"
port "#{ENV['OUTPUT_PORT']}"
scheme "#{ENV['OUTPUT_SCHEME']}"
reload_connections true
logstash_format true
</match>
<match kubernetes.**>
@type record_reformer
remove_keys log,kubernetes
tag mytag.generic
<record>
name ${record['kubernetes']['labels']['app']}
namespace ${record['kubernetes']['namespace_name']}
</record>
</match>
<match kubernetes.**api**>
@type record_reformer
remove_keys log,kubernetes
tag mytag.api
<record>
user_agent ${record['req']['headers']['user-agent']}
</record>
</match>
In such a case it doesn't send the logs for the mytag.generic
I have a very similar use case, and like @embik said, using record_transformer seems like a more fit choice. In my case I'm trying to downcase a key in a json record fluentd is going through, here's the conf -
<filter kubernetes.**.deployment-name**>
@type record_transformer
<record>
level ${record["Level"]}
</record>
</filter>
Hope that helps.