I am looking for a way to collect Java exceptions thrown by containers. I know the function from the logging system of GKE/GCP and would like to implement a similar logging system in our self-hosted cluster.
I am using Prometheus and Grafana for monitoring metrics.
If you are already familiar with the Stackdriver solution from GKE, I'd bet your best choice would be to stick with it and install Stackdriver on your self managed Kubernetes cluster as well:
https://kubernetes.io/docs/tasks/debug-application-cluster/logging-stackdriver/
You need a centralized logging solution. There are some common solutions out there. One of them is the ELK Stack (now named Elastic stack).
It has 3 main components:
Elasticsearch: To store the logs, index them, make them searchable etc.
Logstash: To collect the logs from various sources (containers in your case), parse/filter them and push them to other systems. In ELK's case, push them to Elasticsearch.
Kibana: A web GUI to visualize the data in Elasticsearch, allows searching, creating visual graphs and so on.
See the official page of Elastic stack for more information.
You can also use Fluentd or Fluent Bit instead of Logstash, so it'll be an EFK stack. I personally had pretty good experience with an EFK stack with Fluent Bit.
For another, lighter alternative, you can check out Grafana Loki, which is kind of a logging extension to the popular monitoring setup of Prometheus+Grafana.