We have a crawling engine that we are trialling on Google Kubernetes Engine.
It appears that we are severely network bound when it comes to making request outside the google network.
We have been in-touch with an architect at google, who though that perhaps there was some rate-limiting being applied inside the google data centre. He mentioned that I should raise a support ticket with Google to investigate. Raising a ticket involves subscribing to a support plan (which I am not ready to do until the network issues are addressed) [a bit of a catch-22].
Looking at the network documentation: https://cloud.google.com/network-tiers/?hl=en_US it seems that rates might be severely limited. I'm not sure that I'm reading this right, but are we saying 6Mbps network?
I'm reaching out to the community / Google to see is what we are seeing is expected, if there is any rate limiting and what options there are to increase raw throughput?
You can raise a ticket with Google using the public issue tracker free of charge. In this case, since it's possibly an issue on the Cloud side of things, raising a ticket in this manner will get a Google Engineer looking into this.