GKE Stackdriver not parsing json

12/29/2018

I have a GKE deployment in the default namespace of my cluster writing json out to stdout. For some reason the lines are being picked up as a textPayload (pasted below) instead of a json payload. Stackdriver is receiving the logs but not parsing the log lines. Anyone know why?

{ "textPayload": "{\"timestamp\":\"2018-12-29T04:37:55Z\",\"severity\":\"INFO\",\"httpRequest\":{\"requestMethod\":\"GET\",\"requestUrl\":\"/healthz\",\"status\":\"200\",\"userAgent\":\"kube-probe/1.11+\",\"remoteIp\":\"10.28.5.1:45732\",\"latency\":\"0.000052327s\"},\"message\":\"Completed request\",\"context\":{\"data\":{\"httpRequest\":{\"requestMethod\":\"GET\",\"requestUrl\":\"/healthz\",\"status\":\"200\",\"userAgent\":\"kube-probe/1.11+\",\"remoteIp\":\"10.28.5.1:45732\",\"latency\":\"0.000052327s\"}}}}\n", "insertId": "z8530p0phoyggk6w4", "resource": { "type": "k8s_container", "labels": { "container_name": "charts", "namespace_name": "default", "location": "us-central1", "project_id": "******", "cluster_name": "********", "pod_name": "charts-586d4cb4b6-fmw6m" } }, "timestamp": "2018-12-29T04:37:55.920510496Z", "severity": "INFO", "logName": "projects/******/logs/stdout", "receiveTimestamp": "2018-12-29T04:37:58.865021941Z", "metadata": { "systemLabels": { "provider_zone": "us-central1-b", "top_level_controller_name": "charts", "node_name": "******", "container_image": "*******", "provider_resource_type": "gce_instance", "top_level_controller_type": "Deployment", "name": "charts", "container_image_id": "*******", "service_name": [ "charts-service" ], "provider_instance_id": "*********" }, "userLabels": { "pod-template-hash": "1428076062", "tier": "web", "app": "charts" } } }

What do I need to change so this gets parsed correctly? It's valid json and generated by logrus (and the formater https://github.com/icco/logrus-stackdriver-formatter/blob/sd-v2/formatter.go).

Oh and example log line generated when running locally:

{"timestamp":"2018-12-29T05:07:31.257829Z","severity":"INFO","httpRequest":{"requestMethod":"GET","requestUrl":"/healthz","status":"200","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.28 Safari/537.36","remoteIp":"[::1]:55033","latency":"0.000052380s"},"message":"Completed request","context":{"data":{"httpRequest":{"requestMethod":"GET","requestUrl":"/healthz","status":"200","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.28 Safari/537.36","remoteIp":"[::1]:55033","latency":"0.000052380s"}}}}

-- icco
google-cloud-logging
google-cloud-platform
google-kubernetes-engine
stackdriver

1 Answer

12/29/2018

This is a known bug in GKE: https://issuetracker.google.com/issues/121340431

The fix is to change your cluster version.

-- icco
Source: StackOverflow