I'm currently struggling with the k8s setup for our used elasticsearch with version 2.4.6:
Disable the memory swapping with bootstrap.memory_lock does not work. Memory reservation is failing with the well known error:
[2020-05-22 21:12:22,762][WARN ][bootstrap ] Unable to lock JVM Memory: error=12,reason=Cannot allocate memory
[2020-05-22 21:12:22,764][WARN ][bootstrap ] This can result in part of the JVM being swapped out.
[2020-05-22 21:12:22,765][WARN ][bootstrap ] Increase RLIMIT_MEMLOCK, soft limit: 83968000, hard limit: 83968000
[2020-05-22 21:12:22,765][WARN ][bootstrap ] These can be adjusted by modifying /etc/security/limits.conf, for example:
# allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
I basically tried to follow the guidelines from this source: Memory Settings but it's still not properly running.
Any idea how to solve this issue?
deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: elasticsearch
name: elasticsearch
spec:
replicas: 1
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- image: elasticsearch:2.4.6
imagePullPolicy: ""
name: elasticsearch
env:
- name: ES_JAVA_OPTS
value: "-Xmx512m -Xms512m"
- name: ES_HEAP_SIZE
value: "1g"
- name: bootstrap.memory_lock
value: "true"
ports:
- containerPort: 9200
- containerPort: 9300
volumeMounts:
- mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
subPath: elasticsearch.yml
name: elasticsearch-config
initContainers:
- name: fix-permissions
image: alpine:3.6
command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
securityContext:
privileged: true
volumeMounts:
- name: elasticsearch-data
mountPath: /usr/share/elasticsearch/data
- name: increase-vm-max-map
image: alpine:3.6
command: ["/sbin/sysctl", "-w", "vm.max_map_count=262144"]
securityContext:
privileged: true
- name: fix-ulimit
image: alpine:3.6
command: ["sh", "-c", "ulimit -n 65536"]
securityContext:
privileged: true
hostname: elasticsearch
restartPolicy: Always
serviceAccountName: ""
volumes:
- name: elasticsearch-data
persistentVolumeClaim:
claimName: elasticsearch-data
- name: elasticsearch-config
configMap:
name: elasticsearch-config
Can post configmap and pvc if needed but i think they are not really related to this issue.
You probably can't fix it, it's very possible that Elasticsearch running in a container does not have the required system permissions.
mlockall
is only required if you have swap enabled, which is uncommon on modern systems. If you don't have any swap space configured, don't worry about mlockall
.