I have an application deployed on kubernetes on GKE,
Kubernetes version: v1.7.11-gke.1
Stackdriver Logging
is enabled on my cluster
fluntd-gcp image on my cluster (by default):
gcr.io/google-containers/fluentd-gcp:2.0.9
my logs were all ok, seen in stackdriver, but since a few days ago logs from one deployment (lets call it my-app
) stopped arriving in stackdriver
even though they are logged from my app :
kubectl logs -f my-app-3270987706-cx0r2 --namespace=production
{"time":"2018-01-30 16:11:13.155","msg":"ignoring xml"}
{"time":"2018-01-30 16:11:14.155","msg":"success blabla"}
I see the following logs from fluentd
:
2018-01-30 16:11:46 +0000 [warn]: emit transaction failed:
error_class=Errno::ENOENT error="No such file or directory @ sys_fail2 -
(/var/log/fluentd-buffers/kubernetes.system.buffer..b563203c1da7cb5e1.log, /var/log/fluentd-
buffers/kubernetes.system.buffer..q563203c1da7cb5e1.log)" tag="docker"
2018-01-30 16:11:46 +0000 [warn]: suppressed same stacktrace
2018-01-30 16:11:46 +0000 [error]: Exception emitting record:
No such file or directory @ sys_fail2 -
(/var/log/fluentd-buffers/kubernetes.system.buffer..b563203c1da7cb5e1.log,
/var/log/fluentd-buffers/kubernetes.system.buffer..q563203c1da7cb5e1.log)
why logs arent shipped to stackdriver? how can I fix it?
edit:
Ill note that the logs of other apps do appear in stackdriver
the logs of the failing app are very big - maybe thats why they fail to log?