I want to implement a jaeger installation with persistent storage using elasticsearch like backend on my Kubernetes cluster on Google cloud platform.

I am using the jaeger kubernetes templates and I am starting with elasticsearch production setup.

I've downloaded and modified the configmap.yml file in order to change the password field value and the elasticsearch.yml file in order to fix the password value which I've changed.

My customized .yml files has stayed of this way:

configmap.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: jaeger-configuration
  labels:
    app: jaeger
    jaeger-infra: configuration
data:
  span-storage-type: elasticsearch
  collector: |
    es:
      server-urls: http://elasticsearch:9200
      username: elastic
      password: **my-password-value**
    collector:
      zipkin:
        http-port: 9411
  query: |
    es:
      server-urls: http://elasticsearch:9200
      username: elastic
      password:  **my-password-value**
  agent: |
    collector:
      host-port: "jaeger-collector:14267"

elasticsearch.yml

apiVersion: v1
kind: List
items:
- apiVersion: apps/v1beta1
  kind: StatefulSet
  metadata:
    name: elasticsearch
    labels:
      app: jaeger
      jaeger-infra: elasticsearch-statefulset
  spec:
    serviceName: elasticsearch
    replicas: 1
    template:
      metadata:
        labels:
          app: jaeger-elasticsearch
          jaeger-infra: elasticsearch-replica
      spec:
        containers:
          - name: elasticsearch
            image: docker.elastic.co/elasticsearch/elasticsearch:5.6.0
            imagePullPolicy: Always
            command:
              - bin/elasticsearch
            args:
              - "-Ehttp.host=0.0.0.0"
              - "-Etransport.host=127.0.0.1"
            volumeMounts:
              - name: data
                mountPath: /data
            readinessProbe:
              exec:
                command:
                - curl
                - --fail
                - --silent
                - --output
                - /dev/null
                - --user
                - elastic:**my-password-value**
                - localhost:9200
              initialDelaySeconds: 5
              periodSeconds: 5
              timeoutSeconds: 4
        volumes:
          - name: data
            emptyDir: {}
- apiVersion: v1
  kind: Service
  metadata:
    name: elasticsearch
    labels:
      app: jaeger
      jaeger-infra: elasticsearch-service
  spec:
    clusterIP: None
    selector:
      app: jaeger-elasticsearch
    ports:
    - port: 9200
      name: elasticsearch
    - port: 9300
      name: transport

And then, I've created the kubernetes cluster configuration with the new password value from my machine to my KGE via kubectl command

~/w/j/AddPersistVolumToPods ❯❯❯ kubectl create -f configmap.yml
configmap/jaeger-configuration created
~/w/j/AddPersistVolumToPods ❯❯❯

And I've created the elasticsearch service via StatefulSet specialized pod (also with the new password value) from my machine to my KGE via kubectl command

~/w/j/AddPersistVolumToPods ❯❯❯ kubectl create -f elasticsearch.yml
statefulset.apps/elasticsearch created
service/elasticsearch created
~/w/j/AddPersistVolumToPods ❯❯❯

I can see that I have the elasticsearch service created on my GKE cluster

~/w/j/A/production-elasticsearch ❯❯❯ kubectl get services
NAME            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE
elasticsearch   ClusterIP   None          <none>        9200/TCP,9300/TCP   41m
kubernetes      ClusterIP   10.39.240.1   <none>        443/TCP             1h
~/w/j/A/production-elasticsearch ❯❯❯

And I have the elasticsearch-0 pod which have the docker container of elasticsearch service

~/w/j/A/production-elasticsearch ❯❯❯ kubectl  get pod elasticsearch-0
NAME              READY   STATUS    RESTARTS   AGE
elasticsearch-0   0/1     Running   0          25m
~/w/j/A/production-elasticsearch ❯❯❯

But when I can detail my pod on KGE, I see that my pod have some warnings and is not healthy ...

I get the pod description detail and I get this warning

Warning  Unhealthy              2m6s (x296 over 26m)  kubelet, gke-jaeger-persistent-st-default-pool-d72f7fde-ggrk  Readiness probe failed:

Here, some part of my entire output to describe command

~/w/j/A/production-elasticsearch ❯❯❯ kubectl describe pod elasticsearch-0
Name:           elasticsearch-0
Namespace:      default
Node:           gke-jaeger-persistent-st-default-pool-d72f7fde-ggrk/10.164.0.2
Start Time:     Tue, 08 Jan 2019 13:57:52 +0100
Labels:         app=jaeger-elasticsearch
                controller-revision-hash=elasticsearch-c684bb745
                jaeger-infra=elasticsearch-replica
                statefulset.kubernetes.io/pod-name=elasticsearch-0
Annotations:    kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container elasticsearch
Status:         Running
IP:             10.36.2.7
Controlled By:  StatefulSet/elasticsearch
Containers:
  elasticsearch:
    Container ID:  docker://54d935f3e07ead105464a003745b80446865eb2417da593857d21c56610f704b
    Image:         docker.elastic.co/elasticsearch/elasticsearch:5.6.0
    Image ID:      docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:f95e7d4256197a9bb866b166d9ad37963dc7c5764d6ae6400e551f4987a659d7

                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age                   From                                                          Message
  ----     ------                 ----                  ----                                                          -------
  Normal   Scheduled              27m                   default-scheduler                                             Successfully assigned elasticsearch-0 to gke-jaeger-persistent-st-default-pool-d72f7fde-ggrk
  Normal   SuccessfulMountVolume  27m                   kubelet, gke-jaeger-persistent-st-default-pool-d72f7fde-ggrk  MountVolume.SetUp succeeded for volume "data"
  Normal   SuccessfulMountVolume  27m                   kubelet, gke-jaeger-persistent-st-default-pool-d72f7fde-ggrk  MountVolume.SetUp succeeded for volume "default-token-vkxnj"
  Normal   Pulling                27m                   kubelet, gke-jaeger-persistent-st-default-pool-d72f7fde-ggrk  pulling image "docker.elastic.co/elasticsearch/elasticsearch:5.6.0"
  Normal   Pulled                 26m                   kubelet, gke-jaeger-persistent-st-default-pool-d72f7fde-ggrk  Successfully pulledimage "docker.elastic.co/elasticsearch/elasticsearch:5.6.0"
  Normal   Created                26m                   kubelet, gke-jaeger-persistent-st-default-pool-d72f7fde-ggrk  Created container
  Normal   Started                26m                   kubelet, gke-jaeger-persistent-st-default-pool-d72f7fde-ggrk  Started container
  Warning  Unhealthy              2m6s (x296 over 26m)  kubelet, gke-jaeger-persistent-st-default-pool-d72f7fde-ggrk  Readiness probe failed:
~/w/j/A/production-elasticsearch ❯❯❯

I go to the container log section on GCP and I get the following:

And in the audit log section I can see something like this:

resourceName: "core/v1/namespaces/default/pods/elasticsearch-0"     
response: {
   @type: "core.k8s.io/v1.Status"    
   apiVersion: "v1"    
   code: 500    

details: {…}    
   kind: "Status"    
   message: "The POST operation against Pod could not be completed at this time, please try again."    

metadata: {…}    
   reason: "ServerTimeout"    
   status: "Failure"    
  }
  serviceName: "k8s.io"   

status: {
   code: 13    
   message: "The POST operation against Pod could not be completed at this time, please try again."    
  }
 }

If I try with the original files and I change the password via KGE on GCP I get this error:

Pod "elasticsearch-0" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)

After that I've create a pod, is not possible update or perform some changes?

kubectl apply -f ..... ? ... I suposse

How to can I change the elasticsearch password?

If I want configure a persistent volume claim on this pod, can I perform this before the kubectl create -fcommand and my volume and mountPath will be created on container and KGE?

If somebody can point me in the correct address, their support will be highly appreciated.