K
Q

JSON format for Stackdriver logging in Google Kubernetes Engine

3/14/2018

I'm currently looking for simplest possible JSON log messages that would simply write a severity and a message to Stackdriver logging from a container that is run in Kubernetes Engine and is using the managed Fluentd daemon.

Basically I'm writing single line JSON entries as follows.

{"severity": "DEBUG", "message": "I'm a debug entry"}
{"severity": "ERROR", "message": "I'm an error entry"}

These end up in Stackdriver logging with following results.

  • Severity is always INFO
  • There's JSON payload in the log entry, and the only content is the message, i.e. severity does not go there.

My conclusion is that Fluentd recognizes log row as JSON, but what I don't understand is that how the severity is not set into log entries correctly. Am I e.g. missing some mandatory fields that need to be in place?

-- Iso J
google-cloud-platform
google-kubernetes-engine
kubernetes

Similar Questions

Deploy to IBM Cloud Kubernetes from deployment.yml
Clusters and nodes formation in Kubernetes
Run Automatic Integration Tests on Kubernetes Cluster
adding network flag --network-plugin=cni to kubelet

3 Answers

3/21/2018

My conclusion of the logging behavior is the following as I've now got things working.

  • GKE handles the resource/logName part, no need to worry about this one.
  • You can log any JSON structure, it will go into jsonPlayload
  • Plain text log rows go into plainText, and same applies e.g. to broken JSON. In few occasions I've seen two JSON log rows in a single textPayload, i.e. broken JSON. Whether it's my applications or something else, I don't know yet.
  • There are special JSON fields such as severity that will not go into jsonPayload. Works as documented in GCP logging entry.
-- Iso J
Source: StackOverflow
11/26/2018

an update for logging severity, severity can be logged within the jsonPayload and the fluentd daemon moves it to the outside of jsonPayload. it works with using the string values of severity and not the enums, eg, "severity": "ERROR" inside jsonPayload works fine.

-- Shanmuhanathan Thiagaraja
Source: StackOverflow
3/20/2018

From the information you provided I guess fluentd is passing your whole JSON as as a jsonpayload as a logEntry and providing the logName, resource type and the rest of required information from the environment variables.

In the end what Stackdriver is receiving must look something like this:

{
 "logName": "projects/[YOUR PROJECT ID]/logs/[KUBERNETES LOG]",
 "entries": [
  {
   "jsonPayload": {
    "message": "I'm an ERROR entry",
    "severity": "ERROR"
   },
   "resource": {
    "labels": {
     "project_id": "[YOUR PROJECT ID]",
     "instance_id": "[WHATEVER]",
     "zone": "[YOUR ZONE]"
    },
    "type": "gce_instance"
   }
  }
 ]
}

So you are actually getting the content of the JSON payload on Stackdriver, but as the severity is defined either outside the JSON payload or, if you want to do it inside you'll have to use "severity": enum([NUMERICAL VALUE])

The numerical values of each log level are:

Enums
DEFAULT (0) The log entry has no assigned severity level.
DEBUG (100) Debug or trace information.
INFO (200) Routine information, such as ongoing status or performance.
NOTICE (300) Normal but significant events, such as start up, shut down, or a configuration change.
WARNING (400) Warning events might cause problems.
ERROR (500) Error events are likely to cause problems.
CRITICAL (600) Critical events cause more severe problems or outages.
ALERT (700) A person must take an action immediately.
EMERGENCY (800) One or more systems are unusable.

So, including the field "severity": enum(500) should log the entry as an ERROR instead to fallback to the default INFO.

-- Jordi Miralles
Source: StackOverflow