Before looking at Kubernetes, we are writing all our logs to stdout(according to 12-factor-app) and using logspout to collect the logs to Logstash. And in Logstash we then route logs to different targets:
How these things can be done in Kubernetes? I know that with Heapster you can see JVM level graphs(memory usages, etc) or even maybe Heapster can send events to Riemann in order to alert some system level statistics(e.g., disk is full). But for stuff on the application level, what would be the right approach then?
Heapster should be grabbing the stdout from the containers as well and can send the data to different backends (sinks). It would essentially be an API call with the data. Check out: https://github.com/kubernetes/heapster/blob/master/docs/sink-configuration.md
I'm not 100% sure on stdout being the only method for a 12fa, but we use a in-house logging lib that also streams the stdout to our logging engine (graylog). That happens inside the app so that the log messages are preserved as a full 'event' vs heapster or other stdout scrapings treating each line as an event.