How to get the audit, file beat and metric beat logs in kubernetes cluster using EFK stack?

11/5/2018

I am trying to get the audit, file beat, and metric beat logs together using Fluentd in Kibana dashboard of my kubernetes cluster. I am able to get the audit, file beat and metric beat log separately as specific indexes like filebeat-, auditbeat- and metricbeat-* in my Kibana dashboard.

Could anybody suggest me? Is there any possibility to get the above 3 types of logs within a single index?

-- gayathri
elasticsearch
fluentd
kibana
kubernetes

1 Answer

11/5/2018

Yes, assuming you are talking about an EFK stack and not an ELK stack. In your Fluentd configs you can have something like this:

<match *.**>
  type copy
  <store>
    type elasticsearch
    host localhost
    port 9200
    include_tag_key true
    tag_key @log_name
    logstash_format true
    flush_interval 10s
    index_name fluentd.common.%Y%m%d
  </store>
</match>

They will all go to the same index fluentd.common.%Y%m%d, as opposed to having index_name fluentd.${tag}.%Y%m%d.

-- Rico
Source: StackOverflow