So far in our legacy deployments of webservices to VM clusters, we have effectively been using Log4j2 based multi-file logging on to a persistent Volume where the log files are rolled over each day. We have a need to maintain logs for about 3 months, before they can be purged.

We are migrating to a Kubernetes Infrastructure and have been struggling on what would be the best logging strategy to adapt with Kubernetes Clusters. We don't quite like the strategies involving spitting out all logging to STDOUT/ERROUT and using come centralized tools like Datadog to manage the logs.

Our Design requirements for the Kubernetes Logging Solution are:

1. Using Lo4j2 to multiple files appenders.
2. We want to maintain the multi-file log appender structure.
3. We want to preserve the rolling logs in archives for about 3-months
4. Need a way to have easy access to the logs for searching, filtering etc.
5. The Kubectrl setup for viewing logs may be a bit too cumbersome for our needs.
6. Ideally we would like to use the Datadog dashboard approach BUT using multi-file appenders.
7. The serious limitation of Datadog we run into is the need for having everything pumped to STDOUT.