Please take a look at below example which shows how to make a connection to a service (NodePort) between two private GKE clusters:

This example will use two GKE clusters:

• gke-private-cluster-main - this will be the cluster with a simple hello-app
• gke-private-cluster-europe - this cluster will be able to communicate with the main cluster

To simplify it all the clusters will have one node only.

Create a deployment and a service on gke-private-cluster-main

Below is a simple example of hello-app and a service which will expose hello-app on port 30051:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  selector:
    matchLabels:
      app: hello
      version: 1.0.0
  replicas: 3
  template:
    metadata:
      labels:
        app: hello
        version: 1.0.0
    spec:
      containers:
      - name: hello
        image: "gcr.io/google-samples/hello-app:1.0"
        env:
        - name: "PORT"
          value: "50001"
---
apiVersion: v1
kind: Service
metadata:
  name: hello-service
spec:
  selector:
    app: hello
  ports:
    - name: hello-port
      port: 50001
      targetPort: 50001
      nodePort: 30051
  type: NodePort

Apply it and check the internal IP address of a node which spawned this pods. You can check it with either:

• GCP -> Compute Engine -> VM Instances
• kubectl get nodes -o wide

In my case it was 10.156.0.2

Try to access it from gke-private-cluster-europe

You can SSH to a node of gke-private-cluster-europe and try to invoke command from a node: curl 10.156.0.2:30051. You should be able to communicate with this service and get the output like below:

Hello, world!
Version: 1.0.0
Hostname: hello-5d79ccdb55-vrrbs

To check connectivity from the inside of a pod you would need a image that has already curl builtin. Internet is a place for all kinds of awesome things and in fact there is a image that has curl available. You can spawn a pod with curl with below YAML:

apiVersion: v1
kind: Pod
metadata:
  name: curl
  namespace: default
spec:
  containers:
  - image: curlimages/curl
    command:
      - sleep
      - "infinity"
    imagePullPolicy: IfNotPresent
    name: curl
  restartPolicy: Always

After applying above YAML, you can exec into the pod and check for yourself with below commands:

• $ kubectl exec -it curl -- /bin/sh
• $ curl 10.156.0.2:30051

The output from the inside of a cluster will look like this:

curl: (28) Failed to connect to 10.156.0.2 port 30051: Operation timed out

It worked for a node but it does not work for a pod.

Allow the traffic:

To allow above network connectivity you will need to:

• Open Google Cloud Platform
  • Check the network tag of a gke-private-cluster-main node
    • Go to Compute Engine
    • Find the node of a gke-private-cluster-main
    • Click on it to get more details
    • Copy the network tag which should look similar to: gke-gke-private-cluster-main-80fe50b2-node
  • Check the pod address range of gke-private-cluster-europe:
    • Go to Kubernetes Engine
    • Find your gke-private-cluster-europe
    • Click on it to get more details
    • Copy the pod address range which should look similar to: 10.24.0.0/14

With network tag and pod range copied you can create your firewall rule. Please go to:

VPC Network -> Firewall rules -> Create a firewall rule

Please take a specific look on parts where network tag and pod range ip is used as it will be different for you.

Apply it and check again if a pod in gke-private-cluster-europe can access the 10.156.0.2:30051.

It should give you output below:

Hello, world!
Version: 1.0.0
Hostname: hello-5d79ccdb55-6s8xh

Please let me know if you have any questions in that.