GCP GKE load balancer connectio refused

12/18/2019

I'm doing a deployment on the GKE service and I find that when I try to access the page the message

ERR_CONNECTION_REFUSED

I have defined a load balancing service for deployment and the configuration is as follows.

This is the .yaml for the deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bonsai-onboarding
spec:
  selector:
    matchLabels:
      app: bonsai-onboarding
  replicas: 2
  template:
    metadata:
      labels:
        app: bonsai-onboarding
    spec:
     containers:
     - name: bonsai-onboarding
       image: "eu.gcr.io/diaphanum/onboarding-iocash-master_web:v1"
       ports:
       - containerPort: 3000

This is the service .yaml file.

apiVersion: v1
kind: Service
metadata:
  name: lb-onboarding
spec:
  type: LoadBalancer
  selector:
    app: bonsai-onboarding
  ports:
  - protocol: TCP
    port: 3000
    targetPort: 3000

This working fine, and all is green in GKE :)

kubectl get pods,svc
NAME                                     READY   STATUS    RESTARTS   AGE
pod/bonsai-onboarding-8586b9b699-flhbn   1/1     Running   0          3h23m
pod/bonsai-onboarding-8586b9b699-p9sn9   1/1     Running   0          3h23m

NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)          AGE
service/kubernetes      ClusterIP      XX.xx.yy.YY      <none>         443/TCP          29d
service/lb-onboarding   LoadBalancer   XX.xx.yy.YY   XX.xx.yy.YY   3000:32618/TCP   3h

Then when i tried to connect the error is ERR_CONNECTION_REFUSED

I think is about the network because y did the next test from my local machine

Ping  [load balancer IP]  --->  Correct
Telnet [Load Balancer IP] 3000  --->  Correct

From cloud shell i forward the port 3000 to 8080 and in other cloudShell make a Curl http://localhost:8080, and work fine.

Any idea about the problem?

Thanks in advance

-- Alberto Valencia Carrasco
google-cloud-platform
google-kubernetes-engine
kubernetes
load-balancing

2 Answers

12/18/2019

I've changed a little bit your deployment to check it on my cluster because your image was unreachable:

  • deployment:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: bonsai-onboarding
    spec:
      selector:
        matchLabels:
          app: bonsai-onboarding
      replicas: 2
      template:
        metadata:
          labels:
            app: bonsai-onboarding
        spec:
         containers:
         - name: bonsai-onboarding
           image: nginx:latest
           ports:
           - containerPort: 80
  • service:

    apiVersion: v1
        kind: Service
        metadata:
          name: lb-onboarding
        spec:
          type: LoadBalancer
          selector:
            app: bonsai-onboarding
          ports:
          - protocol: TCP
            port: 3000
            targetPort: 80

and it works out of the box:

kubectl get pods,svc
NAME                                     READY   STATUS    RESTARTS   AGE
pod/bonsai-onboarding-7bdf584499-j2nv7   1/1     Running   0          6m58s
pod/bonsai-onboarding-7bdf584499-vc7kh   1/1     Running   0          6m58s

NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
service/kubernetes      ClusterIP      10.XXX.XXX.1     <none>          443/TCP        8m35s
service/lb-onboarding   LoadBalancer   10.XXX.XXX.230   35.XXX.XXX.235   3000:31637/TCP   67s

and I'm able reach 35.XXX.XXX.235:3000 from any IP:

Welcome to nginx!
...
Thank you for using nginx.

You can check if your app is reachable using this command:

nmap -Pn $(kubectl get svc lb-onboarding -o jsonpath='{.status.loadBalancer.ingress[*].ip}')

Maybe the cause of your problem with "ERR_CONNECTION_REFUSED" in configuration of your image? I found no problem with your deployment and load balancer configuration.

-- Serhii Rohoza
Source: StackOverflow

12/18/2019

Ensure containerPort is defined in the spec of the deployment/statefulset/pod and the application is listening on that port. Also ensure your firewall rules are not blocking the nodeport.

gcloud compute firewall-rules create myservice --allow tcp:3000
-- Paul Dawson
Source: StackOverflow