fluentd isnt shipping logs to stackdriver

1/30/2018

I have an application deployed on kubernetes on GKE,

Kubernetes version: v1.7.11-gke.1

Stackdriver Logging is enabled on my cluster

fluntd-gcp image on my cluster (by default):

gcr.io/google-containers/fluentd-gcp:2.0.9

my logs were all ok, seen in stackdriver, but since a few days ago logs from one deployment (lets call it my-app ) stopped arriving in stackdriver

even though they are logged from my app :

kubectl logs -f my-app-3270987706-cx0r2 --namespace=production

{"time":"2018-01-30 16:11:13.155","msg":"ignoring xml"}
{"time":"2018-01-30 16:11:14.155","msg":"success blabla"}

I see the following logs from fluentd:

2018-01-30 16:11:46 +0000 [warn]: emit transaction failed: 
error_class=Errno::ENOENT error="No such file or directory @ sys_fail2 -
 (/var/log/fluentd-buffers/kubernetes.system.buffer..b563203c1da7cb5e1.log, /var/log/fluentd-
buffers/kubernetes.system.buffer..q563203c1da7cb5e1.log)" tag="docker"

2018-01-30 16:11:46 +0000 [warn]: suppressed same stacktrace

2018-01-30 16:11:46 +0000 [error]: Exception emitting record: 
No such file or directory @ sys_fail2 - 
(/var/log/fluentd-buffers/kubernetes.system.buffer..b563203c1da7cb5e1.log,
 /var/log/fluentd-buffers/kubernetes.system.buffer..q563203c1da7cb5e1.log)

why logs arent shipped to stackdriver? how can I fix it?

edit:

Ill note that the logs of other apps do appear in stackdriver

the logs of the failing app are very big - maybe thats why they fail to log?

-- dina
fluentd
google-cloud-stackdriver
google-kubernetes-engine
kubernetes

0 Answers