My k8s cluster on GKE has a fluentd pod setup by default (currently named fluentd-gcp-v3.1.0-....), but all the logs come out unstructured jammed into a single textPayload field. It would be much nicer to have my nginx (and all other) logs come out in a jsonPayload format.
The Structured Logging docs have a lot of info if you are going to be installing the agent yourself manually, but there doesn't appear to be any information on getting Structured logging enabled on GKE where fluend is installed behind the scenes by default.
Shelling into one of those fluentd containers I see the below files, which are not the default configs (nginx/syslog/apache) listed in the Structured Logging documentation above.
/etc/google-fluentd/config.d
containers.input.conf
monitoring.conf
output.conf
system.input.confSo basically, I've got unstructured nginx logs on GKE/Stackdriver. How to I convert them to structured?
With apologies for the shameless self-promotion... take a look at this:
https://medium.com/google-cloud/customizing-kubernetes-logging-part-1-a1e5791dcda8
This does not [now] require any customization; single-line JSON logs in container STDOUT/STDERR will, according to the Stackdriver/GKE documentation, be parsed as structured data.
https://cloud.google.com/monitoring/kubernetes-engine/legacy-stackdriver/logging#best_practices
Single-line JSON strings written to standard output or standard error will be read into Stackdriver as structured log entries.