How to connect front to back in k8s cluster internal (connection refused)

4/2/2020

Error while trying to connect React frontend web to nodejs express api server into kubernetes cluster.

Can navigate in browser to http:localhost:3000 and web site is ok.

But can't navigate to http:localhost:3008 as expected (should not be exposed)

My goal is to pass REACT_APP_API_URL environment variable to frontend in order to set axios baseURL and be able to establish communication between front and it's api server.

deploy-front.yml

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: gbpd-front
spec:
  selector:
    matchLabels:
      app: gbpd-api
      tier: frontend
      track: stable
  replicas: 2
  template:
    metadata:
      labels:
        app: gbpd-api
        tier: frontend
        track: stable
    spec:
      containers:
        - name: react
          image: binomio/gbpd-front:k8s-3
          ports:
            - containerPort: 3000
          resources:
            limits:
              memory: "150Mi"
            requests:
              memory: "100Mi"
          imagePullPolicy: Always

service-front.yaml

apiVersion: v1
kind: Service
metadata:
  name: gbpd-front
spec:
  selector:
    app: gbpd-api
    tier: frontend
  ports:
  - protocol: "TCP"
    port: 3000
    targetPort: 3000
  type: LoadBalancer

Deploy-back.yaml

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: gbpd-api
spec:
  selector:
    matchLabels:
      app: gbpd-api
      tier: backend
      track: stable
  replicas: 3 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: gbpd-api
        tier: backend
        track: stable
    spec:
      containers:
        - name: gbpd-api
          image: binomio/gbpd-back:dev
          ports:
            - name: http
              containerPort: 3008

service-back.yaml

apiVersion: v1
kind: Service
metadata:
  name: gbpd-api
spec:
  selector:
    app: gbpd-api
    tier: backend
  ports:
  - protocol: "TCP"
    port: 3008
    targetPort: http

I tried many combinations, also tried adding "LoadBalancer" to backservice but nothing...

I can connect perfecto to localhost:3000 and use frontend but frontend can't connect to backend service.

Question 1: What's is the ip/name to use in order to pass REACT_APP_API_URL to fronten correctly? Question 2: Why is curl localhost:3008 not answering?

After 2 days trying almost everything in k8s official docs... can't figure out what's happening here, so any help will be much appreciated.

kubectl describe svc gbpd-api Response:

kubectl describe svc gbpd-api
Name:                     gbpd-api
Namespace:                default
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"gbpd-api","namespace":"default"},"spec":{"ports":[{"port":3008,"p...
Selector:                 app=gbpd-api,tier=backend
Type:                     LoadBalancer
IP:                       10.107.145.227
LoadBalancer Ingress:     localhost
Port:                     <unset>  3008/TCP
TargetPort:               http/TCP
NodePort:                 <unset>  31464/TCP
Endpoints:                10.1.1.48:3008,10.1.1.49:3008,10.1.1.50:3008
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
-- Adolfo Onrubia
cluster-computing
kubernetes
node-jose
reactjs

1 Answer

4/2/2020

I tested your environment, and it worked when using a Nginx image, let's review the environment:

  • The front-deployment is correctly described.
  • The front-service exposes it as loadbalancer, meaning your frontend is accessible from outside, perfect.
  • The back deployment is also correctly described.
  • The backend-service stays with as ClusterIP in order to be only accessible from inside the cluster, great.

Below I'll demonstrate the communication between front-end and back end.

  • I'm using the same yamls you provided, just changed the image to Nginx for example purposes, and since it's a http server I'm changing containerport to 80.

Question 1: What's is the ip/name to use in order to pass REACT_APP_API_URL to fronten correctly?

  • I added the ENV variable to the front deploy as requested, and I'll use it to demonstrate also. You must use the service name to curl, I used the short version because we are working in the same namespace. you can also use the full name: http://gbpd-api.default.svc.cluster.local:3008

Reproduction:

  • Create the yamls and applied them:
$ cat deploy-front.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gbpd-front
spec:
  selector:
    matchLabels:
      app: gbpd-api
      tier: frontend
      track: stable
  replicas: 2
  template:
    metadata:
      labels:
        app: gbpd-api
        tier: frontend
        track: stable
    spec:
      containers:
        - name: react
          image: nginx
          env:
            - name: REACT_APP_API_URL
              value: http://gbpd-api:3008
          ports:
            - containerPort: 80
          resources:
            limits:
              memory: "150Mi"
            requests:
              memory: "100Mi"
          imagePullPolicy: Always

$ cat service-front.yaml 
cat: cat: No such file or directory
apiVersion: v1
kind: Service
metadata:
  name: gbpd-front
spec:
  selector:
    app: gbpd-api
    tier: frontend
  ports:
  - protocol: "TCP"
    port: 3000
    targetPort: 80
  type: LoadBalancer

$ cat deploy-back.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gbpd-api
spec:
  selector:
    matchLabels:
      app: gbpd-api
      tier: backend
      track: stable
  replicas: 3
  template:
    metadata:
      labels:
        app: gbpd-api
        tier: backend
        track: stable
    spec:
      containers:
        - name: gbpd-api
          image: nginx
          ports:
            - name: http
              containerPort: 80

$ cat service-back.yaml 
apiVersion: v1
kind: Service
metadata:
  name: gbpd-api
spec:
  selector:
    app: gbpd-api
    tier: backend
  ports:
  - protocol: "TCP"
    port: 3008
    targetPort: http

$ kubectl apply -f deploy-front.yaml 
deployment.apps/gbpd-front created
$ kubectl apply -f service-front.yaml 
service/gbpd-front created
$ kubectl apply -f deploy-back.yaml 
deployment.apps/gbpd-api created
$ kubectl apply -f service-back.yaml 
service/gbpd-api created
  • Remember, in Kubernetes the communication is designed to be made between services, because the pods are always recreated when there is a change in the deployment or when the pod fail.
$ kubectl get all
NAME                              READY   STATUS    RESTARTS   AGE
pod/gbpd-api-dc5b4b74b-kktb9      1/1     Running   0          41m
pod/gbpd-api-dc5b4b74b-mzpbg      1/1     Running   0          41m
pod/gbpd-api-dc5b4b74b-t6qxh      1/1     Running   0          41m
pod/gbpd-front-66b48f8b7c-4zstv   1/1     Running   0          30m
pod/gbpd-front-66b48f8b7c-h58ds   1/1     Running   0          31m

NAME                 TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)          AGE
service/gbpd-api     ClusterIP      10.0.10.166   <none>         3008/TCP         40m
service/gbpd-front   LoadBalancer   10.0.11.78    35.223.4.218   3000:32411/TCP   42m
  • The pods are the workers, and since they are replaceable by nature, we will connect to a frontend pod to simulate his behaviour and try to connect to the backend service (which is the network layer that will direct the traffic to one of the backend pods).
  • The nginx image does not come with curl preinstalled, so I will have to install it for demonstration purposes:
$ kubectl exec -it pod/gbpd-front-66b48f8b7c-4zstv -- /bin/bash
root@gbpd-front-66b48f8b7c-4zstv:/# apt update && apt install curl -y
done.

root@gbpd-front-66b48f8b7c-4zstv:/# curl gbpd-api:3008
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
  • Now let's try using the environment variable that was defined:
root@gbpd-front-66b48f8b7c-4zstv:/# printenv | grep REACT
REACT_APP_API_URL=http://gbpd-api:3008
root@gbpd-front-66b48f8b7c-4zstv:/# curl $REACT_APP_API_URL
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

Considerations:

Question 2: Why is curl localhost:3008 not answering?

  • Since all yamls are correctly described you must check if image: binomio/gbpd-back:dev is correctly serving on port 3008 as intended.
  • Since it's not a public image, I can't test it, so I'll give you troubleshooting steps:
    • just like we logged inside the front-end pod you will have to log into this backend-pod and test curl localhost:3008.
    • If it's based on a linux distro with apt-get, you can run the commands just like I did on my demo:
    • get the pod name from backend deploy (example: gbpd-api-6676c7695c-6bs5n)
    • run kubectl exec -it pod/<POD_NAME> -- /bin/bash
    • then run apt update && apt install curl -y
    • and test curl localhost:3008
    • if no answer run `apt update && apt install net-tools
    • and test netstat -nlpt, it will have to show you the output of the services running and the respective port, example:
root@gbpd-api-585df9cb4d-xr6nk:/# netstat -nlpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1/nginx: master pro 
  • If the pod does not return nothing even on this approach, you will have to check the code in the image.

Let me know if you need help after that!

-- willrof
Source: StackOverflow