logging: Stackdriver logging not capturing all logs in GKE
We have implemented logging in our flask application which is containerized, all the logs get captured when running in local docker environment and only first few logs are getting captured when using GKE for deployment, both the cases, logging implementation is the exact same. When we checked in the log file within container, all logs are written into it but not getting captured in Stackdriver logging after like 7 to 10 log entires.
Environment details
Steps to reproduce
Logging implemntation inside flask
project_id = retrieve_metadata_server('project/project-id')
cluster_name = retrieve_metadata_server('instance/attributes/cluster-name')
zone = retrieve_metadata_server('instance/attributes/cluster-location')
instance_id = retrieve_metadata_server('instance/id')
resource = Resource(
type='container',
labels={
'cluster_name': cluster_name if cluster_name else '',
'instance_id': instance_id if instance_id else '',
'project_id': project_id if project_id else '',
'zone': zone if zone else '',
}
)
client = gcp_logging.Client()
handler = CloudLoggingHandler(client, name='some-name-for-logger', resource=resource)
app_logger = logging.getLogger('app')
app_logger.addHandler(handler)
ref. image: logs only coming for about 7 to 15 entries in stackdriver, after that logs are getting written to container file but not captured in Stackdriver Logs (ref. to time for each log entries for more info, repeated entries are due to restarting of system)
Thanks!