I have a kubernetes cluster in which I have jenkins and spinnaker pods up and running. I need to implement a logging mechanism which collects and sends logs to splunk server. I chose to do so using fluentd. I have deployed a daemon set of fluentd to run on each node and collect logs from each node and send to splunk server.
It is working fine for logs that we see using "kubectl logs" or logs that come to stdout. However, I need to pick logs from a jenkins job (Console output of a jenkins job build). These logs are not going to std out of the node, and are stored at /var/jenkins_home/jobs/XXX/builds/<buildno> inside the container storage which is not directly accessible to fluentd for log collection.
I am open for any kind of solution to this problem. Please suggest.
Fluentd dont have any feature like this in case of kubernetes as kubernetes will not allow direct access to any third party plugin to read the data from container. For STDOUT log also first processed by kubernetes and kept at node level. After that you will be able to see.
As a workaround you can follow below link.
Long way:
1. Add hostpath directory volume to a jenkins deployment:
https://kubernetes.io/docs/concepts/storage/volumes/#hostpath
2. Run a separate FluentBit/Fluentd to collect logs from that directory.
Both of them support putting path in a dedicated field in a log or to a tag:
https://docs.fluentbit.io/manual/pipeline/inputs/tail
3. Use path/tag to parse-out job name and organize the destination storage accordingly. Or just leave it as is and filter on the view.
Short way: Use jenkins kubernetes plugin to run each job as a separate pod: https://plugins.jenkins.io/kubernetes/. Then it will get collected and tagged by a daemonset separately.