Using a docker image based on this I've created a pod on GKE. The agent will listen for fluentd events via TCP (my other application pods will send events), then forward those logs on to Google Cloud Logging. As is these events are missing some metadata. How can I add this missing information?
(symfony app)--[monolog]-->(google-fluentd-agent)-->(Cloud Logging)
google-fluentd.conf:
<match fluent.**>
type null
</match>
# TCP Connections for fluentd aware applications.
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<match **>
type google_cloud
# Set the chunk limit conservatively to avoid exceeding the GCL limit
# of 10MiB per write request.
buffer_chunk_limit 2M
# Cap the combined memory usage of this buffer and the one below to
# 2MiB/chunk * (24 + 8) chunks = 64 MiB
buffer_queue_limit 24
# Never wait more than 5 seconds before flushing logs in the non-error case.
flush_interval 5s
# Never wait longer than 30 seconds between retries.
max_retry_wait 30
# Disable the limit on the number of retries (retry forever).
disable_retry_limit
</match>
Event in Google Logging with missing data:
{
metadata: {
projectId: "my-project"
serviceName: "container.googleapis.com"
zone: "us-central1-a"
labels: {
container.googleapis.com/cluster_name: "app-staging-a"
compute.googleapis.com/resource_type: "instance"
compute.googleapis.com/resource_name: "cluster-fluentd-1dom0"
container.googleapis.com/instance_id: "296757089355968949"
container.googleapis.com/pod_name: ""
compute.googleapis.com/resource_id: "296757089355968949"
container.googleapis.com/namespace_name: ""
container.googleapis.com/container_name: ""
}
timestamp: "2016-05-16T00:25:37.000Z"
projectNumber: "10568438715"
}
insertId: "94dadf6548d"
log: "symfony.php"
structPayload: {
context: {
stack: [33]
file: "classes.php"
type: 16384
line: 4156
level: 28928
}
level: "INFO"
message: "Using an instance of "This_Function_Method" for function "some_stuff" is deprecated."
}
}
The google_cloud output plugin attempts to parse the namespace, pod, and container names out of the name of the log stream coming into it. In the normal setup, this works because the files on disk coming from each container's stdout/stderr are named that way. In order to get similar parsing behavior, you'll have to similarly craft the log stream name that you send to fluentd (or implement your own logic in a custom fluentd plugin).