Setup

• Google Container Engine (kubernetes)
• Deployment / pod with my web server application (Torando/python)
• Ingress for web server service in kubernetes - which created load balancer in GCP
• Backend service in load balancer with a backend which is the web server
• Frontend which directs to the backend server
• Host and path rules directing custom domain and subdomains to relevant backend
• Firewall rules set as created by ingress

So...

When I created all of the above, I created a new TCP health check with the correct port and everything. After a few minutes that it updates, everything seems to work ok - 3/3 healthy instances in the load balancer and my backend is exposed via the frontend perfectly. I need only TCP health check, not HTTP.

Problem

For some reason, after a few minutes, the health check configuration keeps resetting to the default HTTP health check, that doesn't work, leaving the gateway broken, and any request returns: Error: Server Error The server encountered a temporary error and could not complete your request. Please try again in 30 seconds.

It's very strange, as it seems to work for a few minutes, and only after a while get broken - why???

Thanks