I deployed a Elasticsearch cluster on K8S hosted on EKS nodegroup. The k8s spec file is shown below.
After launching the spec, I can see there are three pods running.
But all of them shows master not discovered
error. Anyone knows what I did wrong in the spec?
{
"type": "server",
"timestamp": "2021-09-21T11:54:40,722Z",
"level": "WARN",
"component": "o.e.c.c.ClusterFormationFailureHelper",
"cluster.name": "elk-cluster",
"node.name": "es-1",
"message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [es-0.es-entrypoint.default.svc.cluster.local] to bootstrap a cluster: have discovered [{es-1}{KXP7FNNqRAeJrsUpMjVZEA}{WQh5hDNvSPaUTp7aYP0WmA}{10.0.2.185}{10.0.2.185:9300}{cdhilmrstw}{ml.machine_memory=15658909696, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {es-0}{BDQHwbS2S06GIfbDZRqTzQ}{fq7y87vfReO_2Ybo0Z29_g}{10.0.1.161}{10.0.1.161:9300}{cdhilmrstw}{ml.machine_memory=15658909696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}, {es-2}{5E2kztdRTOKz529SUblh_Q}{eDfFL66uTEubKfpt-fMeCg}{10.0.1.207}{10.0.1.207:9300}{cdhilmrstw}{ml.machine_memory=15658909696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}]; discovery will continue using [10.0.1.161:9300, 10.0.1.207:9300] from hosts providers and [{es-1}{KXP7FNNqRAeJrsUpMjVZEA}{WQh5hDNvSPaUTp7aYP0WmA}{10.0.2.185}{10.0.2.185:9300}{cdhilmrstw}{ml.machine_memory=15658909696, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"
}
apiVersion: v1
kind: ConfigMap
metadata:
name: es-config
data:
elasticsearch.yml: |
cluster.name: elk-cluster
network.host: "0.0.0.0"
bootstrap.memory_lock: false
discovery.zen.minimum_master_nodes: 2
node.max_local_storage_nodes: 9
cluster.initial_master_nodes:
- es-0.es-entrypoint.default.svc.cluster.local
discovery.seed_hosts:
- es-0.es-entrypoint.default.svc.cluster.local
- es-1.es-entrypoint.default.svc.cluster.local
- es-2.es-entrypoint.default.svc.cluster.local
ES_JAVA_OPTS: -Xms4g -Xmx4g
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: es
namespace: default
spec:
serviceName: es-entrypoint
replicas: 3
selector:
matchLabels:
name: es
volumeClaimTemplates:
- metadata:
name: ebs-claim
spec:
accessModes:
- ReadWriteOnce
storageClassName: ebs-sc
resources:
requests:
storage: 1024Gi
template:
metadata:
labels:
name: es
spec:
volumes:
- name: es-config
configMap:
name: es-config
items:
- key: elasticsearch.yml
path: elasticsearch.yml
initContainers:
- name: permissions-fix
image: busybox
volumeMounts:
- name: ebs-claim
mountPath: /usr/share/elasticsearch/data
command: [ 'chown' ]
args: [ '1000:1000', '/usr/share/elasticsearch/data' ]
containers:
- name: es
image: elasticsearch:7.10.1
resources:
requests:
cpu: 1
memory: 4Gi
ports:
- name: http
containerPort: 9200
- containerPort: 9300
name: inter-node
volumeMounts:
- name: es-config
mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
subPath: elasticsearch.yml
- name: ebs-claim
mountPath: /usr/share/elasticsearch/data
---
apiVersion: v1
kind: Service
metadata:
name: es-entrypoint
spec:
selector:
name: es
ports:
- port: 9200
targetPort: 9200
protocol: TCP
clusterIP: None
Close examine of the error indicated:
...must discover master-eligible nodes es-0.es-entrypoint.default.svc.cluster.local...have discovered [...{es-0}{BDQHwbS2S06GIfbDZRqTzQ}...
The discovered name does not match the wanted master-eligible node name.