Variable number of log streams in Kubernetes

11/24/2021

My application has multiple log streams. And the number of log streams depends on the configuration. Any ideas/pointers on how to handle this on Kubernetes?

I know i can use Using a sidecar container with the logging agent. But this approach requires knowing every log stream in advance, and also couples everything with the app's deployment descriptors.

What if i do this instead?

  • Mount a volume (e.g. /var/log/app/) to my app's pod/container, have my app use this volume to write to as many log files according to configuration
  • Mount the same volume to a daemonset logging agent pod/container, e.g. fluentd, logstash, etc. (assuming it can handle the dynamic structure created by app)

If that is even possible, what would be the drawbacks for such an approach? e.g. "kubectl logs" not working, or any i/o performance issues, etc.? Would i be able to use a persistent volume?

Thanks

-- murtiko
kubernetes
logging

0 Answers