I'm trying to start a simple embedded hazelcast server in a spring boot application on kubernetes. I'm using hazelcast-kubernetes plugin and have followed the documentation here. I tried both LoadBalancer and NodePort service types.

I also tried to deploy the sample application using this example (using the same docker image) and tried both LoadBalancer and NodePort service types.

In both cases I get the below error after 10 retries to connect to Kubernetes Master. Where a.b.c.d is the IP address of the kubernetes master node.

2020-03-09 18:20:57.322  INFO 1 --- [           main] c.h.c.LifecycleService                   : [a.b.c.d]:5701 [dev] [3.12] [a.b.c.d]:5701 is STARTING
2020-03-09 18:20:57.820  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [1] retrying in 1 seconds...
2020-03-09 18:20:59.339  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [2] retrying in 2 seconds...
2020-03-09 18:21:01.713  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [3] retrying in 3 seconds...
2020-03-09 18:21:05.113  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [4] retrying in 5 seconds...
2020-03-09 18:21:10.198  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [5] retrying in 7 seconds...
2020-03-09 18:21:17.807  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [6] retrying in 11 seconds...
2020-03-09 18:21:49.216  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [7] retrying in 17 seconds...
2020-03-09 18:22:06.319  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [8] retrying in 25 seconds...
2020-03-09 18:22:31.963  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [9] retrying in 38 seconds...
2020-03-09 18:23:15.423  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [10] retrying in 57 seconds...
2020-03-09 18:24:23.099 ERROR 1 --- [           main] c.h.i.c.i.DiscoveryJoiner                : [a.b.c.d]:5701 [dev] [3.12] Failure in KubernetesClient
com.hazelcast.kubernetes.KubernetesClientException: Failure in KubernetesClient
        at com.hazelcast.kubernetes.DefaultKubernetesClient.callGet(DefaultKubernetesClient.java:109) ~[hazelcast-kubernetes-1.3.1.jar!/:?]
        at com.hazelcast.kubernetes.DefaultKubernetesClient.endpointsByName(DefaultKubernetesClient.java:88) ~[hazelcast-kubernetes-1.3.1.jar!/:?]
        .
        .
        .       
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [main.jar:?]
Caused by: java.net.UnknownHostException: kubernetes.default.svc
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) ~[?:1.8.0_111]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_111]

In the second case, where I'm using leszko's sample, the ip address it tries to connect to is the IP of the POD.

[190.168.1.194]:5701 [dev] [3.11] Failure in KubernetesClient

190.168.1.194 is the POD's IP address

I added the below env to the deployment yaml, but doesn't look like it is overriding this varaible.

env:
           - name: service-name
             value: "hazelcast-embedded"

Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: null, service-port: 0, service-label: null, service-label-value: true, namespace: default, resolve-not-ready-addresses: false, kubernetes-master: https://kubernetes.default.svc}