How to collect logs from java app (k8s) to fluentd(k8s)

10/8/2019

I have java app in k8s and fluentd (daemonset). In fluentd conf:

 *`<source>
     @type forward
     port 24224
  </source>
  <match **>
  @type stdout
  </match>`*

I am little bit confused. Do I need use fluentd-logger-java lib? I read in docs, that I need add remotehost for fluentd, but here i don't use service in general. How app will send logs to fluentd pods?

Thanks in advance!

-- Donets
fluentd
kubernetes

2 Answers

10/9/2019

If you are just concerned with the live logs then you can try a product built on fluent,Elastic search and kibana ; you can get it https://logdna.com.

Just add a tag and deploy the demonset.

You can try its free trail for some days

-- Tushar Mahajan
Source: StackOverflow

10/9/2019

Given that your Java application can log to stdout and stderr you’ll use fluentd to read that log and, in most cases, ship these logs to a system that can aggregate the logs.

This picture, from the official docs, shows a common pattern of configuring node-level logging in Kubernetes with e.g. fluentd as Pods deployed with a DaemonSet:

enter image description here

In the above picture, the logging-agent will be fluentd and my-pod will be your Pod with a container running your Java app. The Logging Backend, from a fluentd configuration perspective, is optional but of course highly recommended. Basically you can choose to output your logs via fluentd stdout.

For this to function properly fluentd will need read access to the container logs, this is accomplished by mounting the log dir e.g. /var/lib/docker/containers into the fluentd container.

We’ve successfully used this fluentd example ConfigMap, with some modifications to read logs from the nodes and ship them to Elasticsearch. Check out the containers.input.conf part of that ConfigMap for more info on container logs and how to digest them.

Note that you shouldn't need to use the fluentd-logger-java library to start using fluentd, although you could use it as another type of logger in your Java application. Out-of-the-box you should be able to let Java log everything to stdout and stderr and read the logs with fluentd.

-- mikejoh
Source: StackOverflow