Folks,

I'm trying to run gRPC python server with go stub.

It's working properly locally and as dockerized, however, when I'm converting to deployments and run it on kubernetes GKE cluster, I'm getting frequent "connection refused" error as below

I'm client! --> working
Retrieving ...  --> working
id:1 username:"admin" email:"admin@admin.admin"  --> working
id:2 username:"user1"  --> working
I'm client! --> working
Retrieving ...  --> working
2021/11/27 00:55:22 Retrive error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.96.3.78:50051: connect: connection refused"

Also tried to add WithKeepaliveParams() dial option and close connection explicitly at the end without defer but still getting the same error.

Here is my stub code

package main

import (
	"context"
	"fmt"

	// "io"
	"log"
	"time"

	"github.com/nurhun/grpc_django_go_client/accountpb"
	"google.golang.org/grpc"
	"google.golang.org/grpc/keepalive"
)

func main() {
	var kacp = keepalive.ClientParameters{
		Time:                10 * time.Second, // send pings every 10 seconds if there is no activity
		Timeout:             time.Second,      // wait 1 second for ping back
		PermitWithoutStream: true,             // send pings even without active streams
	}

	for {
		fmt.Println("I'm client!")
   		cc, err := grpc.Dial("crud:50051", grpc.WithInsecure(), grpc.WithDisableHealthCheck(),  grpc.WithKeepaliveParams(kacp))
		if err != nil {
			log.Fatal("Connection error", err)
		}
		// defer cc.Close()

		c := accountpb.NewUserControllerClient(cc)

		// ##### Using Retrive with User ID to get user data #####
		fmt.Println("Retrieving ...")
		res1, err := c.Retrieve(context.Background(), &accountpb.UserRetrieveRequest{Id: 1})
		if err != nil {
			log.Fatal("Retrive error: ", err)
		}

		fmt.Println(res1)

		res2, err := c.Retrieve(context.Background(), &accountpb.UserRetrieveRequest{Id: 2})
		if err != nil {
			log.Fatal("Retrive error: ", err)
		}

		fmt.Println(res2)

		time.Sleep(10 * time.Second)

		cc.Close()
	}
}

Any idea where is this error comes from ?