i have 3 nodes in kubernetes cluster - one of them is master. i created elasticsearch by helm instalation with custom values
---
# Permit co-located instances for solitary minikube virtual machines.
antiAffinity: "soft"
# Shrink default JVM heap.
esJavaOpts: "-Xmx128m -Xms128m"
# Allocate smaller chunks of memory per pod.
resources:
requests:
cpu: "100m"
memory: "512M"
limits:
cpu: "1000m"
memory: "512M"
# Request smaller persistent volumes.
volumeClaimTemplate:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 100M
rest of configuration is default. this is kubectl get all
NAME READY STATUS RESTARTS AGE
pod/elasticsearch-master-0 0/1 Running 0 24m
pod/elasticsearch-master-1 0/1 Running 0 24m
pod/elasticsearch-master-2 0/1 Running 0 24m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elasticsearch-master ClusterIP 10.110.224.134 <none> 9200/TCP,9300/TCP 24m
service/elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 24m
service/glusterfs-dynamic-3b22309e-ecb2-4874-9186-1d0d8e16e364 ClusterIP 10.99.52.135 <none> 1/TCP 24m
service/glusterfs-dynamic-5c51d80c-8a33-4240-bc1d-2a578aca6ff9 ClusterIP 10.100.75.99 <none> 1/TCP 24m
service/glusterfs-dynamic-75293711-ee36-4f98-b645-9162e362c4a9 ClusterIP 10.98.220.143 <none> 1/TCP 24m
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 27h
NAME READY AGE
statefulset.apps/elasticsearch-master 0/3 24m
result of describe
one of pods, rest of them was similar
Warning FailedScheduling 25m (x3 over 25m) default-scheduler error while running "VolumeBinding" filter plugin for pod "elasticsearch-master-1": pod has unbound immediate PersistentVolumeClaims
Normal Scheduled 25m default-scheduler Successfully assigned default/elasticsearch-master-1 to kube.node1
Normal Pulling 25m kubelet, kube.node1 Pulling image "docker.elastic.co/elasticsearch/elasticsearch:7.6.0"
Normal Pulled 20m kubelet, kube.node1 Successfully pulled image "docker.elastic.co/elasticsearch/elasticsearch:7.6.0"
Normal Created 20m kubelet, kube.node1 Created container configure-sysctl
Normal Started 20m kubelet, kube.node1 Started container configure-sysctl
Normal Pulled 20m kubelet, kube.node1 Container image "docker.elastic.co/elasticsearch/elasticsearch:7.6.0" already present on machine
Normal Created 20m kubelet, kube.node1 Created container elasticsearch
Normal Started 20m kubelet, kube.node1 Started container elasticsearch
Warning Unhealthy 10s (x124 over 20m) kubelet, kube.node1 Readiness probe failed: Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )
Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )
logs of pods gives this error every few seconds kubectl logs pod/xxxxxxx
{"type": "server", "timestamp": "2020-02-13T17:49:28,299Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "failed to resolve host [elasticsearch-master-headless]",
"stacktrace": ["java.net.UnknownHostException: elasticsearch-master-headless",
"at java.net.InetAddress$CachedAddresses.get(InetAddress.java:798) ~[?:?]",
"at java.net.InetAddress.getAllByName0(InetAddress.java:1489) ~[?:?]",
"at java.net.InetAddress.getAllByName(InetAddress.java:1348) ~[?:?]",
"at java.net.InetAddress.getAllByName(InetAddress.java:1282) ~[?:?]",
"at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:528) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:470) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:801) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) ~[elasticsearch-7.6.0.jar:7.6.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]"] }
I tried change replicas to 1 - pod was runinng with ready 1/1 but problem with resolve host to headless was too. anyone can help
Environment: 3 host with centos8, Kubernetes v1.17.3, storageClass: gluster-provisioner (default) kubernetes.io/glusterfs with heketi api.