Can't set up a Elasticsearch cluster in k8s?

9/21/2021

I deployed a Elasticsearch cluster on K8S hosted on EKS nodegroup. The k8s spec file is shown below.

After launching the spec, I can see there are three pods running.

But all of them shows master not discovered error. Anyone knows what I did wrong in the spec?

{
  "type": "server",
  "timestamp": "2021-09-21T11:54:40,722Z",
  "level": "WARN",
  "component": "o.e.c.c.ClusterFormationFailureHelper",
  "cluster.name": "elk-cluster",
  "node.name": "es-1",
  "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [es-0.es-entrypoint.default.svc.cluster.local] to bootstrap a cluster: have discovered [{es-1}{KXP7FNNqRAeJrsUpMjVZEA}{WQh5hDNvSPaUTp7aYP0WmA}{10.0.2.185}{10.0.2.185:9300}{cdhilmrstw}{ml.machine_memory=15658909696, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {es-0}{BDQHwbS2S06GIfbDZRqTzQ}{fq7y87vfReO_2Ybo0Z29_g}{10.0.1.161}{10.0.1.161:9300}{cdhilmrstw}{ml.machine_memory=15658909696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}, {es-2}{5E2kztdRTOKz529SUblh_Q}{eDfFL66uTEubKfpt-fMeCg}{10.0.1.207}{10.0.1.207:9300}{cdhilmrstw}{ml.machine_memory=15658909696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}]; discovery will continue using [10.0.1.161:9300, 10.0.1.207:9300] from hosts providers and [{es-1}{KXP7FNNqRAeJrsUpMjVZEA}{WQh5hDNvSPaUTp7aYP0WmA}{10.0.2.185}{10.0.2.185:9300}{cdhilmrstw}{ml.machine_memory=15658909696, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"
}
apiVersion: v1
kind: ConfigMap
metadata:
  name: es-config
data:
  elasticsearch.yml: |
    cluster.name: elk-cluster
    network.host: "0.0.0.0"
    bootstrap.memory_lock: false
    discovery.zen.minimum_master_nodes: 2
    node.max_local_storage_nodes: 9
    cluster.initial_master_nodes:
      - es-0.es-entrypoint.default.svc.cluster.local
    discovery.seed_hosts:
      - es-0.es-entrypoint.default.svc.cluster.local
      - es-1.es-entrypoint.default.svc.cluster.local
      - es-2.es-entrypoint.default.svc.cluster.local
  ES_JAVA_OPTS: -Xms4g -Xmx4g
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es
  namespace: default
spec:
  serviceName: es-entrypoint
  replicas: 3
  selector:
    matchLabels:
      name: es
  volumeClaimTemplates:
  - metadata:
      name: ebs-claim
    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: ebs-sc
      resources:
        requests:
          storage: 1024Gi
  template:
    metadata:
      labels:
        name: es
    spec:
      volumes:
        - name: es-config
          configMap:
            name: es-config
            items:
              - key: elasticsearch.yml
                path: elasticsearch.yml
      initContainers:
        - name: permissions-fix
          image: busybox
          volumeMounts:
            - name: ebs-claim
              mountPath: /usr/share/elasticsearch/data
          command: [ 'chown' ]
          args: [ '1000:1000', '/usr/share/elasticsearch/data' ]
      containers:
        - name: es
          image: elasticsearch:7.10.1
          resources:
            requests:
              cpu: 1
              memory: 4Gi
          ports:
            - name: http
              containerPort: 9200
            - containerPort: 9300
              name: inter-node
          volumeMounts:
            - name: es-config
              mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
              subPath: elasticsearch.yml
            - name: ebs-claim
              mountPath: /usr/share/elasticsearch/data
---
apiVersion: v1
kind: Service
metadata:
  name: es-entrypoint
spec:
  selector:
    name: es
  ports:
    - port: 9200
      targetPort: 9200
      protocol: TCP
  clusterIP: None
-- Joey Yi Zhao
amazon-eks
amazon-web-services
elasticsearch
kubernetes

1 Answer

9/22/2021

Close examine of the error indicated:

...must discover master-eligible nodes es-0.es-entrypoint.default.svc.cluster.local...have discovered [...{es-0}{BDQHwbS2S06GIfbDZRqTzQ}...

The discovered name does not match the wanted master-eligible node name.

-- gohm'c
Source: StackOverflow