Minikube - Not able to get any result from elastic search to if it uses existing indices

6/5/2020

I am trying to load my existing local elastic search indices into kubernetes (version - minikube v1.9.2) elastic search pod.

What I finally understood is I have to use mountpath and hostpath combination to do that. Additionlay if I want to provide a custom index file (not the default one), then I have to use a configMap to override path.data of config/elasticsearch.yml

I did those as below and it created a directory in mount path and update config/elasticsearch.yml file but a mount path directory does not contain the host path directory’s content.

I could not figure out the reason behind it. Could some one let me know what am I doing wrong here?

Then went I head and manually copied indexes from local host to kubernetes pod using

kubectl cp localelasticsearhindexdirectory podname:/data/elk/

But then I tried do a elastic search and it gives me a empty result ( even though index manually copied).

If I use the same index with a local elastic search ( not on kubernetes) then I can get the result.

Could someone please give some advice to diagnose following issues

  1. Why mount path does not have the hostpjths content
  2. How to debug / What steps should I follow understand why it’s not able to get the result with the elasticsearch on pod?
kind: Deployment
metadata:
  name: elasticsearch
spec:
  selector:
    matchLabels:
      run: elasticsearch
  replicas: 1
  template:
    metadata:
      labels:
        run: elasticsearch
    spec:
      containers:
        - image: docker.elastic.co/elasticsearch/elasticsearch:6.6.1
          name: elasticsearch
          imagePullPolicy: IfNotPresent
          env:
            - name: discovery.type
              value: single-node
            - name: cluster.name
              value: elasticsearch
          ports:
          - containerPort: 9300
            name: nodes
          - containerPort: 9200
            name: client
          volumeMounts:
          - name: storage
            mountPath: /data/elk
          - name: config-volume
            mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
            subPath: elasticsearch.yml
      volumes:
      - name: config-volume
        configMap:
         name: elasticsearch-config
      - name: storage
        hostPath:
         path: ~/elasticsearch-6.6.1/data     
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: elasticsearch-config
data:
  elasticsearch.yml: |
    cluster:
      name: ${CLUSTER_NAME:elasticsearch-default}

    node:
      master: ${NODE_MASTER:true}
      data: ${NODE_DATA:true}
      name: ${NODE_NAME:node-1}
      ingest: ${NODE_INGEST:true}
      max_local_storage_nodes: ${MAX_LOCAL_STORAGE_NODES:1}

    processors: ${PROCESSORS:1}

    network.host: ${NETWORK_HOST:_site_}

    path:
      data: ${DATA_PATH:"/data/elk"}
      repo: ${REPO_LOCATIONS:[]}

    bootstrap:
      memory_lock: ${MEMORY_LOCK:false}

    http:
      enabled: ${HTTP_ENABLE:true}
      compression: true
      cors:
        enabled: true
        allow-origin: "*"

    discovery:
      zen:
        ping.unicast.hosts: ${DISCOVERY_SERVICE:elasticsearch-discovery}
        minimum_master_nodes: ${NUMBER_OF_MASTERS:1}

    xpack:
      license.self_generated.type: basic            ```


**service.yaml**


```apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  labels:
    service: elasticsearch
spec:
  ports:
    - name: client
      port: 9200
      protocol: TCP
      targetPort: 9200
    - name: nodes
      port: 9300
      protocol: TCP
      targetPort: 9300
  type: NodePort

  selector:
    run: elasticsearch```
-- nantitv
elasticsearch
kubernetes

1 Answer

6/6/2020

Solution in https://stackoverflow.com/questions/38682114/hostpath-with-minikube-kubernetes worked for me. To mount a local directory into a pod in minikube (version - v1.9.2), you have to mount that local directory into minikube then use minikube mounted path in hostpath (https://minikube.sigs.k8s.io/docs/handbook/mount/).

 minikube mount ~/esData:/indexdata
📁  Mounting host path /esData into VM as /indexdata ...
    ▪ Mount type:   <no value>
 User ID:      docker
 Group ID:     docker
    ▪ Version:      9p2000.L
    ▪ Message Size: 262144
    ▪ Permissions:  755 (-rwxr-xr-x)


       ▪ Options:      map[]
        ▪ Bind Address: 192.168.5.6:55230
    🚀  Userspace file server: ufs starting
    ✅  Successfully mounted ~/esData to /indexdata

📌  NOTE: This process must stay alive for the mount to be accessible ...

You have to run minikube mount in a separate terminal because it starts a process and stays there until you unmount.

Instead of doing it as Deployment as in the original question, now I am doing it as Statefulset but the same solution will work for Deployment also.

Another issue which I faced during mounting was elastic search server pod was throwing java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes . Then I saw here that I have to use initContainers to set full permission in /usr/share/elasticsearch/data/nodes.

Please see my final yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
spec:
  serviceName: "elasticsearch"
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      initContainers:
      - name: set-permissions
        image: registry.hub.docker.com/library/busybox:latest
        command: ['sh', '-c', 'mkdir -p /usr/share/elasticsearch/data && chown 1000:1000 /usr/share/elasticsearch/data' ]
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:6.6.1
        env:
        - name: discovery.type
          value: single-node
        ports:
        - containerPort: 9200
          name: client
        - containerPort: 9300
          name: nodes
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      volumes:
      - name: data
        hostPath:
          path: /indexdata
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  labels:
    service: elasticsearch
spec:
  ports:
  - port: 9200
    name: client
  - port: 9300
    name: nodes
  type: NodePort  
  selector:
    app: elasticsearch
-- nantitv
Source: StackOverflow