I have a relatively large number of services running under Kubernetes / Google Container Engine, with a cluster created through the Google Cloud Console UI. In the containers, I have Java services logging in JSON format.

Most of the time this just works, but recently more and more logs have "stopped" at a seemingly random point in time, with the latest entry (at any log level) being from hours or days ago.

I am not changing any parameters and have been unable to isolate what causes this. What's going on here? What can I do to improve and/or troubleshoot this?