I have a fairly simple setup where I have a number of different workers monitoring different resources. When one of them finds something new they publish a message to Google PubSub and then I have a worker pulling the PubSub topic for messages and sending them via Google Cloud Messenger.

This should be really simple and mostly works. However I am running into an interesting situation where when I pull I only sometimes get messages returned, which would make sense if the PubSub topic was empty. However the topic is not empty and keeps growing in size. The messages my worker pulling gets are now usually about 45 minutes late.

According to Stack Driver my PubSub message queue is now at 10,000 messages waiting to be delivered. :(

Here is the code for the worker doing the pulling: https://github.com/Regner/en-notifications

19:05:36.000 INFO:root:Polling for new notifications...
19:05:25.000 INFO:root:Polling for new notifications...
19:05:15.000 INFO:root:Polling for new notifications...
19:05:04.000 INFO:root:Polling for new notifications...
19:04:55.000 INFO:root:Polling for new notifications...
19:04:44.000 INFO:root:Polling for new notifications...
19:04:44.000 INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): gcm-http.googleapis.com
19:04:44.000 INFO:root:Sending message to the following topic "/topics/zkb-kills-1000181"
19:04:44.000 INFO:root:Got message ID 17004238187730 with attributes {u'subtitle': u'Mhorbids One just lost a Executioner worth 3,431,051.31 ISK in Evaulon.', u'service': u'en-zkillboard', u'title': u'zKillboard', u'url': u'https://zkillboard.com/kill/52505094/', u'topic': u'zkb-kills-1000181', u'collapse_key': u'zkb-kills-1000181'}.
19:04:44.000 INFO:root:Polling for new notifications...

Stack driver graphs showing message queue

Some other information: * This is all running as containers withing Google Container Engine * I have tried spinning up more replicas of the pulling container but it doesn't seem to change the rate at which messages get pulled.

I would really appreciate any help in figuring out why this is. It is really confusing me greatly.

Thank you in advance for any help you can offer.

Regner