I have a microservice deployed in a Tomcat container/pod. There are four different files generated in the container - access.log, tomcat.log, catalina.out and application.log (log4j output). What is the best approach to send these logs to Elasticsearch (or similar platform).
I read through the information on this page Logging Architecture - Kubernetes 5. Is “Sidecar container with a logging agent” the best option for my use case?
Is it possible to fetch pod labels (e.g.: version) and add it to each line? If it is doable, use a logging agent like fluentd? (I just want to know the direction I should take).
Yes, the best option for your use case is to have to have one tail -f
sidecar per log file and then install either a fluentd
or a fluent-bit
daemonset that will handle shipping and enriching the log events.
The fluentd elasticsearch cluster addon is available at that link. It will install a fluentd daemonset and a minimal ES cluster. The ES cluster is not production ready so please see the README for details on what must be changed.
Is it possible to fetch pod labels (e.g.: version) and add it to each line?
You can mount information from Pod metadata description to its file system, after that you can configure your agent to use this data. Here is an example:
apiVersion: v1
kind: Pod
metadata:
name: volume-test
spec:
containers:
- name: container-test
image: busybox
volumeMounts:
- name: all-in-one
mountPath: "/projected-volume"
readOnly: true
volumes:
- name: all-in-one
projected:
sources:
- secret:
name: mysecret
items:
- key: username
path: my-group/my-username
- downwardAPI:
items:
- path: "labels"
fieldRef:
fieldPath: metadata.labels
- path: "cpu_limit"
resourceFieldRef:
containerName: container-test
resource: limits.cpu
- configMap:
name: myconfigmap
items:
- key: config
path: my-group/my-config
If it is doable, use a logging agent like fluentd?
Tomcat cannot send logs to Elasticsearch by itself, it needs an agent for that (e.g., Fluentd, Logstash). So, if you want to use Exposing logs directly from the application option, you need to build a Tomcat image with the agent in it. And it seems almost the same as Using a sidecar container with the logging agent option with a harder way to configure. Exposing logs directly from the application option is more related to applications developed by you.