K
Q

Question

How to fix "failed to resolve host [elasticsearch-discovery]"

6/17/2019

Installing elasticsearch using helm charts - https://github.com/helm/charts/tree/master/stable/elasticsearch

State of pods -

augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj   0/1     Running            0          21s
augmented-data-explorer-elasticsearch-data-0                   0/1     Init:0/1           0          21s
augmented-data-explorer-elasticsearch-master-0                 0/1     Init:0/1           0          22s

 kubectl -n zen log augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-06-17T20:14:33,543][INFO ][o.e.e.NodeEnvironment    ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [543.9gb], net total_space [599.7gb], types [rootfs]
[2019-06-17T20:14:33,546][INFO ][o.e.e.NodeEnvironment    ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] heap size [503.6mb], compressed ordinary object pointers [true]
[2019-06-17T20:14:33,551][INFO ][o.e.n.Node               ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] node name [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj], node ID [v5Pb-XSKSBCi-7PLgCO9lQ]
[2019-06-17T20:14:33,552][INFO ][o.e.n.Node               ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] version[6.7.0], pid[1], build[oss/docker/8453f77/2019-03-21T15:32:29.844721Z], OS[Linux/3.10.0-862.14.4.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]
[2019-06-17T20:14:33,552][INFO ][o.e.n.Node               ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-14395150705010143341, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Djava.net.preferIPv4Stack=true, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=docker]
[2019-06-17T20:14:36,014][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [aggs-matrix-stats]
[2019-06-17T20:14:36,014][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [analysis-common]
[2019-06-17T20:14:36,014][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [ingest-common]
[2019-06-17T20:14:36,014][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [ingest-geoip]
[2019-06-17T20:14:36,014][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [ingest-user-agent]
[2019-06-17T20:14:36,014][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [lang-expression]
[2019-06-17T20:14:36,015][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [lang-mustache]
[2019-06-17T20:14:36,015][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [lang-painless]
[2019-06-17T20:14:36,015][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [mapper-extras]
[2019-06-17T20:14:36,015][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [parent-join]
[2019-06-17T20:14:36,015][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [percolator]
[2019-06-17T20:14:36,015][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [rank-eval]
[2019-06-17T20:14:36,015][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [reindex]
[2019-06-17T20:14:36,015][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [repository-url]
[2019-06-17T20:14:36,015][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [transport-netty4]
[2019-06-17T20:14:36,015][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] loaded module [tribe]
[2019-06-17T20:14:36,016][INFO ][o.e.p.PluginsService     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] no plugins loaded
[2019-06-17T20:14:44,222][INFO ][o.e.d.DiscoveryModule    ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] using discovery type [zen] and host providers [settings]
[2019-06-17T20:14:45,823][INFO ][o.e.n.Node               ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] initialized
[2019-06-17T20:14:45,823][INFO ][o.e.n.Node               ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] starting ...
[2019-06-17T20:14:46,518][INFO ][o.e.t.TransportService   ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] publish_address {10.1.213.210:9300}, bound_addresses {0.0.0.0:9300}
[2019-06-17T20:14:46,528][INFO ][o.e.b.BootstrapChecks    ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-06-17T20:14:46,639][WARN ][o.e.d.z.UnicastZenPing   ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] failed to resolve host [augmented-data-explorer-elasticsearch-discovery]
java.net.UnknownHostException: augmented-data-explorer-elasticsearch-discovery: Name or service not known
    at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
    at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929) ~[?:?]
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1515) ~[?:?]
    at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848) ~[?:?]
    at java.net.InetAddress.getAllByName0(InetAddress.java:1505) ~[?:?]
    at java.net.InetAddress.getAllByName(InetAddress.java:1364) ~[?:?]
    at java.net.InetAddress.getAllByName(InetAddress.java:1298) ~[?:?]
    at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:536) ~[elasticsearch-6.7.0.jar:6.7.0]
    at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:489) ~[elasticsearch-6.7.0.jar:6.7.0]
    at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:737) ~[elasticsearch-6.7.0.jar:6.7.0]
    at org.elasticsearch.discovery.zen.UnicastZenPing.lambda$resolveHostsLists$0(UnicastZenPing.java:189) ~[elasticsearch-6.7.0.jar:6.7.0]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-6.7.0.jar:6.7.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-06-17T20:14:49,723][WARN ][o.e.d.z.ZenDiscovery     ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2019-06-17T20:14:49,724][WARN ][o.e.d.z.UnicastZenPing   ] [augmented-data-explorer-elasticsearch-client-5dd945bc4-qdrjj] failed to resolve host [augmented-data-explorer-elasticsearch-discovery]

I do not see any endpoints defined -

NAME                                              ENDPOINTS                                                              AGE
ade-db2                                                                                                                  5m25s
augmented-data-explorer                           10.1.213.241:5000                                                      5m25s
augmented-data-explorer-elasticsearch-client                                                                             5m25s
augmented-data-explorer-elasticsearch-discovery                                                                          5m25s

From the error it cannot resolve augmented-data-explorer-elasticsearch-discovery.

What do I need to define for ping.unicast.hosts, currently I have this -

discovery:
      zen:
        ping.unicast.hosts: ${DISCOVERY_SERVICE:}
        minimum_master_nodes: ${MINIMUM_MASTER_NODES:2}

Also I see in the master svc there is no IP and endpoint -

[root@dv-demo4-master-1 ~]# kubectl -n zen describe svc augmented-data-explorer-elasticsearch-discovery
Name:              augmented-data-explorer-elasticsearch-discovery
Namespace:         zen
Labels:            app=elasticsearch
                   chart=elasticsearch-1.28.0
                   component=master
                   heritage=Tiller
                   release=augmented-data-explorer
Annotations:       <none>
Selector:          app=elasticsearch,component=master,release=augmented-data-explorer
Type:              ClusterIP
IP:                None
Port:              <unset>  9300/TCP
TargetPort:        transport/TCP
Endpoints:         
Session Affinity:  None
Events:            <none>

Also in the describe elasticsearch master pod I see no IP for readiness check -

  elasticsearch:
    Container ID:   
    Image:          docker.elastic.co/elasticsearch/elasticsearch-oss:6.7.0
    Image ID:       
    Port:           9300/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:  1
    Requests:
      cpu:      25m
      memory:   512Mi
    Readiness:  http-get http://:9200/_cluster/health%3Flocal=true delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      NODE_DATA:                   false
      DISCOVERY_SERVICE:           augmented-data-explorer-elasticsearch-discovery
      PROCESSORS:                  1 (limits.cpu)
      ES_JAVA_OPTS:                -Djava.net.preferIPv4Stack=true -Xms512m -Xmx512m  
      EXPECTED_MASTER_NODES:       1
      MINIMUM_MASTER_NODES:        1
      RECOVER_AFTER_MASTER_NODES:  1
    Mounts:
      /usr/share/elasticsearch/config/elasticsearch.yml from config (rw)
      /usr/share/elasticsearch/data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from augmented-data-explorer-elasticsearch-master-token-6xtlz (ro)

How do I resolve this issue?

-- Nidhi

elasticsearch

kubernetes-helm

K
Q

How to fix "failed to resolve host [elasticsearch-discovery]"

Similar Questions

0 Answers

KQ

How to fix "failed to resolve host [elasticsearch-discovery]"

Similar Questions

0 Answers

K
Q