kubenetes elasticsearch - failed to resolve host elasticsearch-master-headless

2/13/2020

i have 3 nodes in kubernetes cluster - one of them is master. i created elasticsearch by helm instalation with custom values

---
# Permit co-located instances for solitary minikube virtual machines.
antiAffinity: "soft"

# Shrink default JVM heap.
esJavaOpts: "-Xmx128m -Xms128m"

# Allocate smaller chunks of memory per pod.
resources:
  requests:
    cpu: "100m"
    memory: "512M"
  limits:
    cpu: "1000m"
    memory: "512M"

# Request smaller persistent volumes.
volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 100M

rest of configuration is default. this is kubectl get all

NAME                         READY   STATUS    RESTARTS   AGE
pod/elasticsearch-master-0   0/1     Running   0          24m
pod/elasticsearch-master-1   0/1     Running   0          24m
pod/elasticsearch-master-2   0/1     Running   0          24m

NAME                                                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/elasticsearch-master                                     ClusterIP   10.110.224.134   <none>        9200/TCP,9300/TCP   24m
service/elasticsearch-master-headless                            ClusterIP   None             <none>        9200/TCP,9300/TCP   24m
service/glusterfs-dynamic-3b22309e-ecb2-4874-9186-1d0d8e16e364   ClusterIP   10.99.52.135     <none>        1/TCP               24m
service/glusterfs-dynamic-5c51d80c-8a33-4240-bc1d-2a578aca6ff9   ClusterIP   10.100.75.99     <none>        1/TCP               24m
service/glusterfs-dynamic-75293711-ee36-4f98-b645-9162e362c4a9   ClusterIP   10.98.220.143    <none>        1/TCP               24m
service/kubernetes                                               ClusterIP   10.96.0.1        <none>        443/TCP             27h

NAME                                    READY   AGE
statefulset.apps/elasticsearch-master   0/3     24m

result of describe one of pods, rest of them was similar

 Warning  FailedScheduling  25m (x3 over 25m)    default-scheduler    error while running "VolumeBinding" filter plugin for pod "elasticsearch-master-1": pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled         25m                  default-scheduler    Successfully assigned default/elasticsearch-master-1 to kube.node1
  Normal   Pulling           25m                  kubelet, kube.node1  Pulling image "docker.elastic.co/elasticsearch/elasticsearch:7.6.0"
  Normal   Pulled            20m                  kubelet, kube.node1  Successfully pulled image "docker.elastic.co/elasticsearch/elasticsearch:7.6.0"
  Normal   Created           20m                  kubelet, kube.node1  Created container configure-sysctl
  Normal   Started           20m                  kubelet, kube.node1  Started container configure-sysctl
  Normal   Pulled            20m                  kubelet, kube.node1  Container image "docker.elastic.co/elasticsearch/elasticsearch:7.6.0" already present on machine
  Normal   Created           20m                  kubelet, kube.node1  Created container elasticsearch
  Normal   Started           20m                  kubelet, kube.node1  Started container elasticsearch
  Warning  Unhealthy         10s (x124 over 20m)  kubelet, kube.node1  Readiness probe failed: Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )
Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )

logs of pods gives this error every few seconds kubectl logs pod/xxxxxxx

{"type": "server", "timestamp": "2020-02-13T17:49:28,299Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "failed to resolve host [elasticsearch-master-headless]",
"stacktrace": ["java.net.UnknownHostException: elasticsearch-master-headless",
"at java.net.InetAddress$CachedAddresses.get(InetAddress.java:798) ~[?:?]",
"at java.net.InetAddress.getAllByName0(InetAddress.java:1489) ~[?:?]",
"at java.net.InetAddress.getAllByName(InetAddress.java:1348) ~[?:?]",
"at java.net.InetAddress.getAllByName(InetAddress.java:1282) ~[?:?]",
"at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:528) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:470) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:801) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]"] }

I tried change replicas to 1 - pod was runinng with ready 1/1 but problem with resolve host to headless was too. anyone can help

Environment: 3 host with centos8, Kubernetes v1.17.3, storageClass: gluster-provisioner (default) kubernetes.io/glusterfs with heketi api.

-- drajkry
elasticsearch
kubernetes
kubernetes-helm

0 Answers