Stackdriver logging not capturing all logs in GKE

12/10/2019

logging: Stackdriver logging not capturing all logs in GKE

We have implemented logging in our flask application which is containerized, all the logs get captured when running in local docker environment and only first few logs are getting captured when using GKE for deployment, both the cases, logging implementation is the exact same. When we checked in the log file within container, all logs are written into it but not getting captured in Stackdriver logging after like 7 to 10 log entires.

Environment details

  1. Stackdriver logging in python flask app
  2. Hosted in GKE Cluster with: Stackdriver Logging API - Write Only Stackdriver Monitoring API - Full Stackdriver Trace - Write Only
  3. Python 3.7 with Pipenv
  4. google-api-core v1.14.3
  5. google-auth v1.7.1
  6. google-cloud-core v1.0.3
  7. google-cloud-logging v1.14.0
  8. googleapis-common-protos v1.6.0

Steps to reproduce

  1. deploy the same app in GKE, first few log entries get captured in stackdriver logging but after like 7 to 10 entries, nothing is showing in Stackdriver even though the log file inside the container has all the entries. This happens only if we deploy in GKE, if we are running locally, this logging works perfectly fine.

Logging implemntation inside flask

project_id = retrieve_metadata_server('project/project-id')
cluster_name = retrieve_metadata_server('instance/attributes/cluster-name')
zone = retrieve_metadata_server('instance/attributes/cluster-location')
instance_id = retrieve_metadata_server('instance/id')

resource = Resource(
    type='container',
    labels={
        'cluster_name': cluster_name if cluster_name else '',
        'instance_id': instance_id if instance_id else '',
        'project_id': project_id if project_id else '',
        'zone': zone if zone else '',
    }
)
client = gcp_logging.Client()
handler = CloudLoggingHandler(client, name='some-name-for-logger', resource=resource)
app_logger = logging.getLogger('app')
app_logger.addHandler(handler)

ref. image: logs only coming for about 7 to 15 entries in stackdriver, after that logs are getting written to container file but not captured in Stackdriver Logs (ref. to time for each log entries for more info, repeated entries are due to restarting of system) logs only coming for about 7 to 15 entries in stackdriver

Thanks!

-- Thaiseer
google-cloud-logging
google-cloud-stackdriver
google-kubernetes-engine
logging
python

0 Answers