What I did is follow the instructions in the section of "Build high availability and high bandwidth NAT gateways" in https://cloud.google.com/vpc/docs/special-configurations, and after I tagged all instances with "no-ip", I was no longer able to access the pod logs using "kubectl logs".
Is it because kubectl logs under the hood using ssh? Is there any workaround to see the pod log?
It uses the kube-proxy to connect to the nodes and from there look at the logs in the containers as say docker logs <container-name> would do it. Then it proxies the output back to wherever you are running kubectl from.
Kubernetes uses a lot of iptables and routing in all its nodes so any change that you'd make to the instances where it's running would affect how the components talk to each other. Check if that "no-ip" tag made it change/add/remove firewall rules where your Kubernetes nodes are running.
Hope it helps!
as Patrick mentioned,You need a new route rule that will help all traffic to master node, through the default gateway rather than NAT. Please follow this NAT configuration specifically for GKE which mentions the additional route.
kubectl logs interacts with the k8s api-server on your master, the master then retrieves logs from the container (effectively running docker logs on the node) and sends the info back to you. Anytime you run a kubectl command it interacts with the API server on the master. The master then connects to the node using SSH. You can also find the IP of the master by running kubectl get endpoint. The "kubernetes" endpoint is the IP for your master's endpoint.