Kubernetes and Docker: how to let two service to communicate correctly

1/18/2019

I have two Java microservices (caller.jar which calls called.jar)

We can set the caller service http port through an env-var CALLERPORT and the address of called service through an env-var CALLEDADDRESS. So caller uses two env-var.

We must set also called service env-var CALLEDPORT in order to set the specific http port on which called service is listening for http requests.

I don't know exactly how to simply expose these variables from a Dockerfile, in order to set them using Kubernetes.

Here is how I made the two Dockerfiles:

Dockerfile of caller

FROM openjdk:8-jdk-alpine

# ENV CALLERPORT     (it's own port)
# ENV CALLEDADDRESS  (the other service address)

ADD caller.jar /

CMD ["java", "-jar", "caller.jar"]

Dockerfile of called

FROM openjdk:8-jdk-alpine

# ENV CALLEDPORT (it's own port)

ADD called.jar /

CMD ["java", "-jar", "called.jar"]

With these I've made two Docker images:

  • myaccount/caller
  • myaccount/called

Then I've made the two deployments.yaml in order to let K8s deploy (on minikube) the two microservices using replicas and loadbalancers.

deployment-caller.yaml

apiVersion: v1
kind: Service              
metadata:
  name: caller-loadbalancer
spec:
  type: LoadBalancer       
  ports:
  - port: 8080               
    targetPort: 8080        
  selector:            
    app: caller    
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: caller
  labels:
    app: caller
spec:
  replicas: 2                                             
  minReadySeconds: 15
  strategy:
    type: RollingUpdate                                   
    rollingUpdate: 
      maxUnavailable: 1                                   
      maxSurge: 1                                         
  selector:
    matchLabels:
      app: caller
      tier: caller
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: caller
        tier: caller
    spec:
      containers:
      - image: myaccount/caller
        name: caller
        env:
        - name: CALLERPORT
          value: "8080"
        - name: CALLEDADDRESS
          value: called-loadbalancer  # WHAT TO PUT HERE?!
        ports:
        - containerPort: 8080
          name: caller

And deployment-called.yaml

apiVersion: v1
kind: Service              
metadata:
  name: called-loadbalancer
spec:
  type: LoadBalancer       
  ports:
  - port: 8081               
    targetPort: 8081        
  selector:            
    app: called    
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: called
  labels:
    app: called
spec:
  replicas: 2                                             
  minReadySeconds: 15
  strategy:
    type: RollingUpdate                                   
    rollingUpdate: 
      maxUnavailable: 1                                   
      maxSurge: 1                                         
  selector:
    matchLabels:
      app: called
      tier: called
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: called
        tier: called
    spec:
      containers:
      - image: myaccount/called
        name: called
        env:
        - name: CALLEDPORT
          value: "8081"
        ports:
        - containerPort: 8081
          name: called

IMPORTANT: The single services work well if called singularly (such as calling an healthcheck endpoint) but, when calling the endpoint which involves the communication between the two services, then there is this error:

java.net.UnknownHostException: called

The pods are correctly running and active, but i guess the problem is the part of deployment.yaml in which I must define how to find the pointed service, so here:

spec:
  containers:
    - image: myaccount/caller
      name: caller
      env:
        - name: CALLERPORT
          value: "8080"
        - name: CALLEDADDRESS
          value: called-loadbalancer  # WHAT TO PUT HERE?!
        ports:
        - containerPort: 8080
          name: caller

Neither

called

nor

called-loadbalancer

nor

http://caller 


kubectl get pods,svc -o wide

NAME                          READY     STATUS    RESTARTS   AGE       IP           NODE       NOMINATED NODE   READINESS GATES
pod/called-855cc4d89b-4gf97   1/1       Running   0          3m23s     172.17.0.4   minikube   <none>           <none>
pod/called-855cc4d89b-6268l   1/1       Running   0          3m23s     172.17.0.5   minikube   <none>           <none>
pod/caller-696956867b-9n7zc   1/1       Running   0          106s      172.17.0.6   minikube   <none>           <none>
pod/caller-696956867b-djwsn   1/1       Running   0          106s      172.17.0.7   minikube   <none>           <none>

NAME                          TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE       SELECTOR
service/called-loadbalancer   LoadBalancer   10.99.14.91    <pending>     8081:30161/TCP   171m      app=called
service/caller-loadbalancer   LoadBalancer   10.107.9.108   <pending>     8080:30078/TCP   65m       app=caller
service/kubernetes            ClusterIP      10.96.0.1      <none>        443/TCP          177m      <none>

works if put in that line of the deployment.yaml. So what to put in this line?

-- Alex Mawashi
docker
environment-variables
kubernetes
load-balancing
microservices

3 Answers

1/18/2019

The short answer is you don't need to expose them in the Dockerfile. You can set any environment variables you want when you start a container and they don't have to be specified upfront in the Dockerfile.

You can verify this by starting a container using 'docker run' with '-e' to set env vars and '-it' to get an interactive session. The echo the value of your env var and you'll see it is set.

You can also get a terminal session with one of the containers in your running kubernetes Pod with 'kubectl exec' (https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/). From there you can echo environment variables from there to see that they are set. You can see them more quickly with 'kubectl describe pod ' after getting the pod name with 'kubectl get pods'.

Since you are having problems, you also want to check whether your services are working correctly. Since you are using minikube you can do 'minikube service ' to check they can be accessed externally. You'll also want to check the internal access - see Accessing spring boot controller endpoint in kubernetes pod

Your approach of using service names and ports is valid. With a bit of debugging you should be able to get it working. Your setup is similar to an illustration I did in https://dzone.com/articles/kubernetes-namespaces-explained so referring to that might help (except you are using env vars directly instead of through a configmap but it amounts to the same).

I think that in the caller you are injecting the wrong port in the env var - you are putting the caller's own port and not the port of what it is trying to call.

-- Ryan Dawson
Source: StackOverflow

1/18/2019

First of all - it's completely impossible to understand what do you want. Your post starts from:

We can set...

We must set...

Nobody here doesn't know what do you want to do and it could be much more useful to see some definition of done you are expecting.

This having been said now, I have to turn to your substantive question...

  env:
    - name: CALLERPORT
      value: "8080"
    - name: CALLEDADDRESS
      value: called-loadbalancer  # WHAT TO PUT HERE?!
    ports:
    - containerPort: 8080
      name: caller

This things will be exported by k8s automatically. For example, i have service kibana with a port:80 in service definition:

svc/kibana                                 ClusterIP      10.222.81.249    <none>                              80/TCP              1y        app=kibana

this is how I can get this within the different pod which is in the same namespace:

root@some-pod:/app# env | grep -i kibana
KIBANA_SERVICE_PORT=80
KIBANA_SERVICE_HOST=10.222.81.249

Moving forward, why do you use LoadBalancer? Without any cloud it will be the similar to NodePort, but seems like ClusterIP is all you need. Next, service ports can be the same and there won't be any port collisions, just because ClusterIP is unique every time and therefore socket will be unique for each service. Your services could be described like this:

apiVersion: v1
kind: Service              
metadata:
  name: caller-loadbalancer
spec:
  type: LoadBalancer       
  ports:
  - port: 80 <--------------------           
    targetPort: 8080        
  selector:            
    app: caller  

apiVersion: v1
kind: Service              
metadata:
  name: called-loadbalancer
spec:
  type: LoadBalancer       
  ports:
  - port: 80 <------------------             
    targetPort: 8081        
  selector:            
    app: called

That would simplify using service names just by names without ports specifying:

http://caller-loadbalancer.default.svc.cluster.local
http://called-loadbalancer.default.svc.cluster.local

or

http://caller-loadbalancer.default
http://called-loadbalancer.default

or (within the similar namespace):

http://caller-loadbalancer
http://called-loadbalancer

or (depending on the lib)

caller-loadbalancer
called-loadbalancer

Same things about containerPort/targetPort! Why do you use 8081 and 8080? Who cares about internal container ports? I agree different cases happen, but in this case you have a single process inside and you are definitely not going to run some more processes there, are you? So they also could be the same.

I'd like to advise you to use stackoverflow different way. Do not ask how to do something your way, much better to ask how to do something the best way

-- Konstantin Vustin
Source: StackOverflow

1/18/2019
-- Alex Pliutau
Source: StackOverflow