I've set up a service in kubernetes. However, I cannot reach it over TCP. What are the next debugging steps?

• I've checked the service logs, the process has started
• I've inspected the service endpoints -- it correctly finds the pod
• I've execed bash into the service pod, however, I found Internet connection slow within service (I've checked the node IO, and it isn't the highest that's ever been). My plan was installing telnet and checking can I manually connect to service gRPC server.
• I've tried putting "0.0.0.0/50051" and "[::]:50051" as server address/port with the same results -- not working
• I've tried port-forward with the localhost and service, but it isn't working when connecting via port-forwards
• The service is, of course, tested locally, before deploying in the cloud. It works as expected.
• Kubernetes is installed on google cloud platform via their supplied engine.

For the kubernetes yaml I've setup for TCP service, and I've pretty much copy/paste/modify for another service exposing JSON REST api server. What are some next steps I can take into the debugging?

EDIT 1:

https://pastebin.com/PweEv6gc This is my kubernetes-helm config. I'm trying to reach it within the cluster. When I try to reach it outside the cluster I'm using kubectl port-forward 50051 50051 utility.

EDIT 2:

I've tried to spin up ordinary flask web server and it works with specific pod/k8s/port configuration. This issues is related to grpc. or it's port binding.

I've used:

server.add_insecure_port("0.0.0.0:50051")
server.start()

And managed to connect via node port (10.132.0.25:31128, which I got from kubectl get service oracle but within the cluster it says unreachable.

EDIT 3:

I've tried the same script inside the kubernetes and my localhost.

With using 10.132.0.25:31158 (nodeport): On my localhost it normally connects to the gRPC server. Inside k8s, it doesn't work

with using oracle:50051 Inside k8s it doesn't work

Update: Found the bug....it was hard. The HTTP proxy was set up, and on the tested pod I've used HTTP proxy which probably doesn't implement http/2 (tinyproxy). How can I print gRPC debug logs to get this info?

Update2: https://github.com/grpc/grpc/blob/master/doc/environment_variables.md

for gRPC better debug logs