As per the official es docs, disabling swapping is one of the best performance boosts available to Elasticsearch.
However, it's proving to be difficult to configure. I've spent a number of hours researching and attempting different methods to disable swapping using the official ES docker image on Kubernetes.
When setting bootstrap.memory_lock: true
as an env variable, the image fails to boot up with the error: Unable to lock JVM Memory: error=12, reason=Cannot allocate memory. This can result in part of the JVM being swapped out. Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
. As the docs point out, this is kind of expected. I've even mounted a custom /etc/security/limits.conf
with the settings, but that's failed.
What is the recommended way to disable swapping when using the official es image on k8s?
And, here are the relevant sections of my yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: elastic-data
spec:
serviceName: elastic-data
replicas: 1
template:
spec:
securityContext:
runAsUser: 0
fsGroup: 0
containers:
- name: elastic-data
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.4.0
env:
- name: ES_JAVA_OPTS
value: "-Xms2g -Xmx2g"
- name: cluster.name
value: "elastic-devs"
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: discovery.zen.ping.unicast.hosts
value: "elastic-master.default.svc.cluster.local"
- name: node.master
value: "false"
- name: node.ingest
value: "false"
- name: node.data
value: "true"
- name: network.host
value: "0.0.0.0"
- name: path.data
value: /usr/share/elasticsearch/data
- name: indices.memory.index_buffer_size
value: "512MB"
- name: bootstrap.memory_lock
value: "true"
resources:
requests:
memory: "3Gi"
limits:
memory: "3Gi"
ports:
- containerPort: 9300
name: transport
- containerPort: 9200
name: http
volumeMounts:
- name: data-volume
mountPath: /usr/share/elasticsearch/data
- name: swappiness-config
mountPath: /etc/security/limits.conf
subPath: limits.conf
volumes:
- name: data-volume
persistentVolumeClaim:
claimName: pvc-es
- name: swappiness-config
configMap:
name: swappiness-config
items:
- key: limits.conf
path: limits.conf
limits.conf
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
elasticsearch hard nofile 65536
elasticsearch soft nofile 65536
Which image-type are you using? If its Container-Optimized OS (cos) try to switch to Ubuntu based images
I think, the ulimits in my yaml weren't being recognized, so I followed this post and created an image with a custom entrypoint that set the settings.