Elasticsearch high level rest client, connection reset error in Kubernetes

10/14/2019

I am using a single node elasticsearch server and a Java application based on elasticsearch high level rest client. Both are running in a Kubernetes cluster.

    @Bean(destroyMethod = "close")
    public RestHighLevelClient client(){
      RestHighLevelClient client = null;
      Logger.getLogger(getClass().getName()).info("Connecting to elasticsearch on host : " + host);
      client = new RestHighLevelClient(RestClient.builder(new HttpHost(host, port, "http")));
      return client;
    }

This is working fine until service kept idle for about 10 minutes. When trying to query elasticsearch server an exception is thrown form java service

java.io.IOException: Connection reset
    at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:948) ~[elasticsearch-rest-client-6.4.3.jar!/:7.2.0]
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227) ~[elasticsearch-rest-client-6.4.3.jar!/:7.2.0]
    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1448) ~[elasticsearch-rest-high-level-client-7.2.0.jar!/:7.2.0]
    at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1418) ~[elasticsearch-rest-high-level-client-7.2.0.jar!/:7.2.0]
    at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1388) ~[elasticsearch-rest-high-level-client-7.2.0.jar!/:7.2.0]
    at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:930) ~[elasticsearch-rest-high-level-client-7.2.0.jar!/:7.2.0]

When I send the requests three time to the service it will again works. But after about 10 minutes of idle time service will give the same exception. I have a docker-compose setup with same images but there is no issue like this.

My elasticsearch deployment

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
spec:
  type: NodePort
  ports:
  - name: client
    port: 9200
    targetPort: 9200
  - name: nodes
    port: 9300
    targetPort: 9300
  selector:
    app: elasticsearch

---

apiVersion: apps/v1 
kind: StatefulSet
metadata:
  name: elasticsearch
spec:
  serviceName: elasticsearch
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      nodeSelector:
        beta.kubernetes.io/os: linux
      containers:
      - image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
        name: elasticsearch
        env:
        - name: cluster.name
          value: "docker-cluster"
        - name: 'ES_JAVA_OPTS'
          value: "-Xms512m -Xmx512m"
        - name: discovery.type
          value: "single-node"
        ports:
        - containerPort: 9200
        - containerPort: 9300
          name: mysql
        volumeMounts:
        - name: elasticsearch-persistent-storage
          mountPath: /usr/share/elasticsearch/data
      volumes:
      - name: elasticsearch-persistent-storage
        persistentVolumeClaim:
          claimName: elasticsearch-claim
      initContainers:
      - image: alpine:3.6
        command: ["/sbin/sysctl", "-w", "vm.max_map_count=262144"]
        name: elasticsearch-init
        securityContext:
          privileged: true

My Java Service

    apiVersion: v1
kind: Service
metadata:
  name: search
spec:
  ports:
  - port: 9099
    targetPort: 9099
  selector:
    app: search

---

apiVersion: apps/v1 
kind: Deployment
metadata:
  name: search
spec:
  selector:
    matchLabels:
      app: search
  strategy:
    type: Recreate
  replicas: 1
  template:
    metadata:
      labels:
        app: search
    spec:
      nodeSelector:
        beta.kubernetes.io/os: linux
      containers:
      - image: search-service:0.0.1-SNAPSHOT
        name: search
        env:
        - name: ELASTIC_SEARCH_HOST
          value: elasticsearch
        - name: ELASTIC_SEARCH_PORT
          value: "9200"
        - name: ELASTIC_SEARCH_CLUSTER
          value: docker-cluster
        ports:
        - containerPort: 9099
-- Gayan Viranka
elasticsearch
kubernetes
rest-client

0 Answers