Elasticsearch Service Unavailable on Kibana Dashboard

9/19/2017

I am running Kubernetes Cluster and I found out that when I'm trying to access Kibana Dashboard, I'm getting States "RED" screen and I can see that the elastic search service unavailable. I checked the logs on one of the elastic-search pods and I saw following logs,

[2017-09-19 08:54:33,776][WARN ][transport.netty          ] [Dominus] exception caught on transport layer [[id: 0xf27712f3]], closing connection
java.net.NoRouteToHostException: No route to host
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
    at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
    at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2017-09-19 08:54:44,832][WARN ][rest.suppressed          ] path: /_bulk, params: {}
ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/2/no master];]
    at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:158)
    at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:144)
    at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:204)
    at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:151)
    at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:71)
    at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:149)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85)
    at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
-- Actung
elasticsearch
google-cloud-platform
kibana
kubectl
kubernetes

1 Answer

9/25/2017

Okay I would like to post an answer to my own question, so I was getting queue buffer error on elasticsearch node. I assume that it exceeded the heap memory and hence started failing. I restarted Kibana and the master elasticsearch and it worked like a charm. :)

-- Actung
Source: StackOverflow