We are running java based microservices and we have following scenario

• Application writes debug.log file to /opt/tomcat/logs/debuglog/debug.log and size of log file is 1GB/hour
• Tomcat to write catalina.out , localhost_access_log and localhost.log and size of log is 1GB/hour for all of them

Question is how to solve this problem when we have a lot of logs to centralised and analysed. We are running 20 instances of this application. We have got 150GB logs in flat filess. Followings are the problems,

• Store the logs for 3 years in GCS as per our SLA
• Parse these logs and store them in BQ for bigdata for 1 year
• Parse these logs and store them in ELK for 7 days for developers to analyse any running issue

We are trying to evaluate following,

• As kubernetes recommends to run sidecars for application logs, we may endup running 3 sidecars considering catalina.out will go to stdout. We can use Stack-driver to process the logs and put them to GCS. Problem we see is container explosion specifically with auto scaling. Other problem is to parse the logs from stackdriver to BigQuery or ELK.
• Mount GCS in containers and write there itself. Problem is the GCS is community driven and not production ready. We still have to write solution to parse these logs again
• Use external drive mount to Minion and volume mount to container. Run 1 container per VM to process the logs for different pipelines and scenarios. This is solving a few problems for us like : Logs will not be lost when downscaled, No container explosion , single responsibly container to process different pipelines, move the logs to GCS as per availability. Problem we see is to manage the SSD storage attached to each VM upon up-scale and down-scale.

Any suggestions are welcomed.

EDIT

We end up using custom pipeline on GCP where applications are pushing logs to pub/sub and dataflow is responsible to aggregate and transform the information.