Kubernetes elasticserach health check fails - but only in some containers

5/12/2020

I've got a very strange networking problem trying to get elasticsearch working on a local Kubernetes cluster, and I'm completely stumped on what could be causing the issue. At this point, I don't think this is an Elasticsearch problem, I think there is something odd going on in the host machine, but I can't for the life of me figure out what it is.

TLDR version: "curl -X GET http://127.0.0.1:9200" works from inside some containers, but not other others.

The details are as follows:

I have a 4 node Kubernetes cluster for testing on two different machines.

  • Both hosts have the same operating system (OpenSuse Leap 15.1)
  • They both have the same version of VirtualBox and Vagrant.
  • They both have a 4 node Kubernetes cluster created from the same Vagrantfile, using the same version of the same Vagrant base box (bento/centos-7).
  • Since the Vagrant boxes are the same, both of my environments will have the same version of Docker in the VMs.
  • I've installed Elasticsearch to each cluster using the same Helm chart, and they both use the same Elasticsearch Docker image.

But from there, I run into problems in one of my environments when I do the following:

  1. I run kubectl get pods -A -o wide to find out where the elasticsearch master is running.
  2. I run vagrant ssh to that node.
  3. As root, I run docker ps to find out the id of the container running elasticsearch.
  4. As root, I run docker exec -u root -it container_name /bin/bash to get a shell in the container.
  5. I run curl -X GET http://127.0.0.1:9200/_cluster/health, which is what Kubernetes is using for a health check. In one environment, I get back JSON. In the other, I get "Connection refused"

I can't figure out why the same docker image running in the same kind of virtual machine would produce a different result on a different host.

Can anyone shed some light on this situation?

-- Steve Saliman
docker
elasticsearch
kubernetes
vagrant
virtualbox

0 Answers