K
Q

Give multiple container names in logs pattern to scrape data from

8/19/2021

I have setup EFK stack in K8s cluster. Currently fluentd is scraping logs from all the containers.

I want it to only scrape logs from containers A, B, C and D.

If I had some prefix with as A-app I could do something like below.

"fluentd-inputs.conf": "# HTTP input for the liveness and readiness probes
		<source>
		  @type http
		  port 9880
		</source>
		# Get the logs from the containers running in the node
		<source>
		  @type tail
		  path /var/log/containers/*-app.log // what can I put here for multiple different containers
		  # exclude Fluentd logs
		  exclude_path /var/log/containers/*fluentd*.log
		  pos_file /opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos
		  tag kubernetes.*
		  read_from_head true
		  <parse>
		    @type json
		  </parse>
		</source>
		# enrich with kubernetes metadata
		<filter kubernetes.**>
		  @type kubernetes_metadata
		</filter>
-- confusedWarrior
efk
fluent
kubernetes

Similar Questions

Deserializing ObjectMeta Regardless of Kind
How can I run multiple network interfaces on a k8s node?
How to change request path in kubernetes
How to Backup/Restore TRAINS-server when moving from AMI to local machine
Helm delete release and clear associated storage

1 Answer

8/20/2021

To scrape logs only from specific Pods, you can use:

path /var/log/containers/POD_NAME_1*.log,/var/log/containers/POD_NAME_2*.log,.....,/var/log/containers/POD_NAME_N*.log

To scrape logs from specific containers in specific Pods, you can use:

path /var/log/containers/POD_NAME_1*CONTAINER_NAME*.log,/var/log/containers/POD_NAME_2*CONTAINER_NAME*.log,.....,/var/log/containers/POD_NAME_N*CONTAINER_NAME*.log

I've created a simple example to illustrate how it works.

To scrape logs from web-1 container from app-1 Pod and logs from all containers from app-2 Pod, you can use:

path /var/log/containers/app-1*web-1*.log,/var/log/containers/app-2*.log



$ kubectl logs -f fluentd-htwn5
...
2021-08-20 13:37:44 +0000 [info]: #0 starting fluentd worker pid=18 ppid=7 worker=0
2021-08-20 13:37:44 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/app-1_default_web-1-ae672aa1405b91701d130da34c54ab3106a8fc4901897ebbf574d03d5ca64eb8.log
2021-08-20 13:37:44 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/app-2-64c99b9f5b-tm6ck_default_nginx-cd1bd7617f04000a8dcfc1ccd01183eafbce9d0155578d8818b27427a4062968.log
2021-08-20 13:37:44 +0000 [info]: #0 [in_tail_container_logs] following tail of /var/log/containers/app-2-64c99b9f5b-tm6ck_default_frontend-1-e83acc9e7fc21d8e3c8a733e10063f44899f98078233b3238d6b3dc0903db560.log
2021-08-20 13:37:44 +0000 [info]: #0 fluentd worker is now running worker=0
...
-- matt_j
Source: StackOverflow