How do I configure Eureka (& Kubernetes) so that a microservice (pod) can register to to the eureka server (pod)?

5/6/2020

The problem:
The country microservice can't register to the eureka server (the eureka dashboard doesn't show the country service) when using kubernetes.

The eureka server is configured on localhost:1111. The country service eureka configuration is configured on port 2222 and setup to look for localhost:1111/eureka to reach eureka server. The respective deployments have containerPort setup to 1111 and 2222 respectively. The respective kubernetes Service-s have the NodePorts 30080 and 30081 respectively.

The question:
How do I configure Eureka & Kubernetes so that the country service registers to the eureka server and eureka server keep track of the country service?

In detail

More context:
The eureka registration server app configuration:

...
# Configure this Discovery Server
eureka:
  instance:
    hostname: localhost
...
server:
  port: 1111   # HTTP (Tomcat) port
...

The country (micro)service / app eureka configuration:

...
# HTTP Server
server.port: 2222   # HTTP (Tomcat) port
...
eureka:
  client:
    serviceUrl:
      defaultZone: http://${registration.server.hostname}:${registration.server.port}/eureka/
...
# Eureka server assumed to be on localhost:
registration.server.hostname: localhost
registration.server.port: 1111

Dockerfile has:

EXPOSE 1111
EXPOSE 2222

The application works (country microservice can register at eureka server and eureka server keeps track of country microservice) by running it on my local machine or by using docker-toolbox, without an issue.

I apply the reggo deployment:

...
metadata:
  name: reggo
spec:
  ...
  template:
    ...
    spec:
     containers:
     - name: reggo
       ...
       ports:
       - containerPort: 1111

When I check the log of the pod the Registration Server starts properly.

The reggo kubernetes Service:

...
metadata:
  name: reggo
  ...
spec:
  type: NodePort
  ports:
    - port: 1111
      nodePort: 30080
      name: reggo-port
  ...

From browser I can reach the Eureka dashboard by doing:

  • minikube:30080
  • [minikubeNodeIP]:30080

[minikubeNodeIP] can be found from kubectl describe node.

I apply the country deployment next:

...
metadata:
  name: country
spec:
  ...
  template:
    ...
    spec:
     containers:
     - name: country
       ...
       ports:
       - containerPort: 2222

When I check the log of the pod the Registration Server starts properly.

The reggo kubernetes Service:

...
metadata:
  name: country
  ...
spec:
  type: NodePort
  ports:
    - port: 2222
      nodePort: 30081
      name: country-port
  ...

I get an expected output when I try:

  • minikube:30081/countries/getAll
  • [minikubeNodeIP]:30081/countries/getAll

localhost:1111 and localhost:2222 don't work any more from the browser as for kubernetes I need a Service to access them, which is understandable.

More problem insights:

The log for the country pod has:

WARN RetryableEurekaHttpClient - Request execution failed with message: java.net.ConnectException: Connection refused (Connection refused)

ERROR DiscoveryClient - DiscoveryClient_COUNTRY-SERVICE/country-7b9687b56c-pnx8f:country-service:2222 - was unable to send heartbeat!

When I try to check the communication from within the pods I have the following:
Pods can see expected output when using their localhost and local (target)ports:

$ kubectl exec -it reggo-6f6cf5cfc9-gnzh8 -- curl localhost:1111
{...expected output...}
$ kubectl exec -it country-5c49fb76c-9t86v -- curl localhost:2222
{...expected output...}

Pods don't seem to be able to communicate to the other pod by referencing through localhost and respective (target)port (since localhost translates differently for each pod):

$ kubectl exec -it reggo-6f6cf5cfc9-gnzh8 -- curl localhost:2222
curl: (7) Failed to connect to localhost port 2222: Connection refused
command terminated with exit code 7
$ kubectl exec -it country-5c49fb76c-9t86v -- curl localhost:1111
curl: (7) Failed to connect to localhost port 1111: Connection refused
command terminated with exit code 7

They don't seem to reach each other by communicating as external entity (by using [minikubeNodeIP]:[ServiceNodePort] combination):

$ kubectl exec -it reggo-6f6cf5cfc9-gnzh8 -- curl 192.168.99.101:30081
curl: (7) Failed to connect to 192.168.99.101 port 30081: Connection refused
command terminated with exit code 7
$ kubectl exec -it country-5c49fb76c-9t86v -- curl 192.168.99.101:30080
curl: (7) Failed to connect to 192.168.99.101 port 30080: Connection refused
command terminated with exit code 7

According to the kubernetes cluster networking,

pods on a node can communicate with all pods on all nodes without NAT

so the pods should be able to find each other.

What I have tried

I tried setting up kubernetes Service per pod as per Stefan's suggestion with no luck (check below).

Setting up Eureka with hostname=minikube and ports 30080 & 30081 instead of 1111 & 2222

The Eureka pod couldn't even reach it's own app (Eureka dashboard) from within the pod:

$ kubectl exec -it reggo-5c64bcdc55-dlpxg -- curl minikube:1111
curl: (6) Could not resolve host: minikube
command terminated with exit code 6
$ kubectl exec -it reggo-5c64bcdc55-dlpxg -- curl 192.168.99.101:1111                                                                                                           curl: (7) Failed to connect to 192.168.99.101 port 1111: Connection refused
command terminated with exit code 7
$ kubectl exec -it reggo-5c64bcdc55-dlpxg -- curl 192.168.99.101:30080
curl: (7) Failed to connect to 192.168.99.101 port 30080: Connection refused
command terminated with exit code 7
$ kubectl exec -it reggo-5c64bcdc55-dlpxg -- curl 192.168.99.101:8761
curl: (7) Failed to connect to 192.168.99.101 port 8761: Connection refused
command terminated with exit code 7

Setting up Eureka with hostname=localhost and ports 30080 & 30081 instead of 1111 & 2222

Country service still not registered, though I could reach the Eureka dashboard from within the pods:

$ kubectl exec -it reggo-5c64bcdc55-dnj8b -- curl localhost:30080  
$ kubectl exec -it country-5bc9c7b767-wt748 -- curl localhost:30080

All kubectl exec -it country-5bc9c7b767-wt748 -- curl commands end up with "Connection refused" except countryServiceIP:30081 that ends up with "Connection timed out".
The country kubernetes Service has for Endpoints.
I can't reach any host:port combination from the browser (host being: minikube, [minikubeNodeIP], localhost; port being: 1111, 2222, 30080, 30081).

-- despot
docker
kubernetes
microservices
minikube
netflix-eureka

0 Answers