Deploying Elasticsearch and Kibana was easy using the official helm chart. Now I want to protect it using authentication. This should be no problem any more since Security was open sourced in May 2019 and is avaliable for free. So I enabled xpack security and also added encrypted certificate authentication since it's a requirement.

My values yaml file looks like this:

# Elastic repo blocked by proxy
image: "elasticsearch"
imageTag: "7.6.1"

ingress:
  enabled: true
  tls:
    - hosts:
      - elasticsearch-test.k8s.internal
  paths: ['/']
  hosts:
    - elasticsearch-test.k8s.internal

# Also possible with secrets: https://github.com/elastic/helm-charts/blob/master/elasticsearch/examples/config/values.yaml
extraEnvs:
  - name: ELASTIC_PASSWORD
    value: testpw
  - name: ELASTIC_USERNAME
    value: testuser

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs

Based on an example in the official Helm chart, I create a lightly modified one that generates the certificates and store them as Kubernetes secrets:

STACK_VERSION := 7.6.1
ELASTICSEARCH_IMAGE := elasticsearch:$(STACK_VERSION)
ELASTIC_USER := testuser
ELASTIC_PASSWORD := testpw
HELM_NAME := elasticsearch-test
NAMESPACE := elasticsearch-test

install:
    helm install --name $(HELM_NAME) elastic/elasticsearch --namespace $(NAMESPACE) -f elasticsearch-values.yml

secrets:
    docker rm -f elastic-helm-charts-certs || true
    rm -f elastic-certificates.p12 elastic-certificate.pem elastic-stack-ca.p12 || true

    docker run --name elastic-helm-charts-certs -i -w /app \
        $(ELASTICSEARCH_IMAGE) \
        /bin/sh -c " \
            elasticsearch-certutil ca --out /app/elastic-stack-ca.p12 --pass '' && \
            elasticsearch-certutil cert --name security-master --dns security-master --ca /app/elastic-stack-ca.p12 --pass '' --ca-pass '' --out /app/elastic-certificates.p12"
    docker cp elastic-helm-charts-certs:/app/elastic-certificates.p12 ./ && \
    docker rm -f elastic-helm-charts-certs
    openssl pkcs12 -nodes -passin pass:'' -in elastic-certificates.p12 -out elastic-certificate.pem && \
    kubectl create secret generic elastic-certificates --from-file=elastic-certificates.p12 && \
    kubectl create secret generic elastic-certificate-pem --from-file=elastic-certificate.pem && \
    kubectl create secret generic elastic-credentials  --from-literal=password=$(ELASTIC_PASSWORD) --from-literal=username=$(ELASTIC_USER)
    rm -f elastic-certificates.p12 elastic-certificate.pem elastic-stack-ca.p12

purge:
    kubectl delete secrets elastic-credentials elastic-certificates elastic-certificate-pem || true
    helm del --purge $(HELM_NAME)

clean:
    $(MAKE) purge

But the connection between the Elasticsearch cluster member failed:

{"type": "server", "timestamp": "2020-04-14T14:40:54,038Z", "level": "WARN", "component": "o.e.t.TransportService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Transport response handler not found of id [1]" }
{"type": "server", "timestamp": "2020-04-14T14:40:54,162Z", "level": "WARN", "component": "o.e.t.TransportService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Transport response handler not found of id [2]" }
{"type": "server", "timestamp": "2020-04-14T14:40:56,679Z", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "failed to join {elasticsearch-master-2}{0frrBAvhS_Swe6RFqarh3A}{f4eYyU4ARKSz3kXvNKc4ig}{10.42.3.52}{10.42.3.52:9300}{dilm}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticsearch-master-0}{-F2BUfJLSyCggyDg7uo6pg}{N9KOsW7DSK-fvrj6nJIMEA}{10.42.5.194}{10.42.5.194:9300}{dilm}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=4, lastAcceptedTerm=3, lastAcceptedVersion=22, sourceNode={elasticsearch-master-0}{-F2BUfJLSyCggyDg7uo6pg}{N9KOsW7DSK-fvrj6nJIMEA}{10.42.5.194}{10.42.5.194:9300}{dilm}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}, targetNode={elasticsearch-master-2}{0frrBAvhS_Swe6RFqarh3A}{f4eYyU4ARKSz3kXvNKc4ig}{10.42.3.52}{10.42.3.52:9300}{dilm}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}}]}", 
"stacktrace": ["org.elasticsearch.transport.NodeNotConnectedException: [elasticsearch-master-2][10.42.3.52:9300] Node not connected",
"at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:191) ~[elasticsearch-7.6.1.jar:7.6.1]",
"at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:618) ~[elasticsearch-7.6.1.jar:7.6.1]",
"at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:590) [elasticsearch-7.6.1.jar:7.6.1]",
"at org.elasticsearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:279) [elasticsearch-7.6.1.jar:7.6.1]",
"at org.elasticsearch.cluster.coordination.JoinHelper.sendJoinRequest(JoinHelper.java:212) [elasticsearch-7.6.1.jar:7.6.1]",
"at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$2(JoinHelper.java:136) [elasticsearch-7.6.1.jar:7.6.1]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) [x-pack-security-7.6.1.jar:7.6.1]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) [elasticsearch-7.6.1.jar:7.6.1]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306) [x-pack-security-7.6.1.jar:7.6.1]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:378) [x-pack-security-7.6.1.jar:7.6.1]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:186) [x-pack-security-7.6.1.jar:7.6.1]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130) [x-pack-security-7.6.1.jar:7.6.1]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248) [x-pack-security-7.6.1.jar:7.6.1]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310) [x-pack-security-7.6.1.jar:7.6.1]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321) [x-pack-security-7.6.1.jar:7.6.1]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245) [x-pack-security-7.6.1.jar:7.6.1]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) [x-pack-security-7.6.1.jar:7.6.1]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) [x-pack-security-7.6.1.jar:7.6.1]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) [x-pack-security-7.6.1.jar:7.6.1]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313) [x-pack-security-7.6.1.jar:7.6.1]",
"at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.6.1.jar:7.6.1]",
"at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) [elasticsearch-7.6.1.jar:7.6.1]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.1.jar:7.6.1]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]"] }

What I already tried

• Setting protocol: https
• Added the following to esConfig.elasticsearch.yml:

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12

• Verified that xpack security itself seem to be enabled: curl 'http://localhost:9200/_cat/plugins?v&pretty' in one Elasticsearch pod throws a security_exception: missing authentication credentials for REST request [/_cat/plugins?v&pretty]

Docker images

Elastic has it's own Docker registry on docker.elastic.co/elasticsearch/elasticsearch. But it's blocked in our corporate network, so I tried the official elasticsearch image on the Docker hub. Since it's official and up2date, I assume that those are the same images as on elastics own repo. I couldn't find any information about differences or other reasons why the official hub images shouldn't be used.