Expose kibana (eck operator) via istio gateway - https problem (Error 503)

1/8/2020

I want to expose kibana with istio gateway, for that i used this config files

apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: monitoring
  namespace: {{ .Values.kibana.namespace }}  
spec:
  version: 7.4.2
  http:
    tls:
      selfSignedCertificate:
        disabled: true
    service:
      spec:
        ports:
          - name: http
            port: 9200
            targetPort: 9200  
  nodeSets:
  - name: default
    count: 1
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          env:
          - name: ES_JAVA_OPTS
            value: -Xms1g -Xmx1g
          resources:
            requests:
              memory: 1Gi
              cpu: 0.5
            limits:
              memory: 2Gi
              cpu: 2    
      metadata:
        annotations:
          traffic.sidecar.istio.io/excludeOutboundPorts: "9300"
          traffic.sidecar.istio.io/excludeInboundPorts: "9300"

apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
  name: monitoring
  namespace: {{ .Values.kibana.namespace }}  
spec:
  version: 7.4.2
  count: 1
  elasticsearchRef:
    name: monitoring
  http:
    tls:
      selfSignedCertificate:
        disabled: true
  podTemplate:
    spec:
      containers:
      - name: kibana
        env:
          - name: SERVER_BASEPATH
            value: ""      
        resources:
          requests:
            memory: 1Gi
            cpu: 0.5
          limits:
            memory: 2Gi
            cpu: 2 
    metadata:
      annotations:
        sidecar.istio.io/rewriteAppHTTPProbers: "true"

kind: Gateway
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: kibana-gateway
  namespace: elastic-system
spec:
  servers:
    - hosts:
        - kibana.<domain>
      port:
        name: http2
        number: 80
        protocol: HTTP2
    - hosts:
        - kibana.<domain>
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        credentialName: kibana-tls-secret
        mode: SIMPLE
  selector:
    istio: ingressgateway

kind: VirtualService
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: kibana-rule
  namespace: elastic-system
spec:
  hosts:
    - kibana.<domain>
  gateways:
    - kibana-gateway
  http:
    - match:
        - uri:
            prefix: /
      route:
        - destination:
            host: monitoring-kb-http.elastic-system.svc.cluster.local
            port:
              number: 5601

All other applications that i expose work well using the same configuration but with kibana i have http 503 with this message "upstream connect error or disconnect/reset before headers. reset reason: connection termination"

If i delete https configuration from the gateway and i keep only http , it works. Certificate are generated from cert manager and they work well for other applications (for now i use self signed certificates for test purpose)

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  selfSigned: {}  

{{ $root := . }}
{{- range list "dashboard" "grafana" "prometheus" "tracing" "kiali" "kibana"}}

{{- if index $root.Values .  }}
{{- if index $root.Values . "tls" }}
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: {{ . }}-certificate
  namespace: istio-system
spec:
  secretName: {{ . }}-tls-secret
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: {{ ( index $root.Values . "host" ) | quote }} 
  dnsNames:
  - {{ ( index $root.Values . "host" ) | quote }} 
  acme:
    config:
    - dns01:
         provider: poc-dns
      domains:
      - {{ ( index $root.Values . "host" ) | quote }} 
{{- end }}
{{- end }} 

{{- end }}  

What did you expect to see?

Kibana exposed via https

What did you see instead? Under which circumstances?

503 error with this message "upstream connect error or disconnect/reset before headers. reset reason: connection termination"

Environment

ECK version:

https://download.elastic.co/downloads/eck/1.0.0-beta1/all-in-one.yaml

Istio version: =1.4.0

Kubernetes information:

Cloud: EKS Kubernetes Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.8", GitCommit:"211047e9a1922595eaa3a1127ed365e9299a6c23", GitTreeState:"clean", BuildDate:"2019-10-15T12:11:03Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-c0eccc", GitCommit:"c0eccca51d7500bb03b2f163dd8d534ffeb2f7a2", GitTreeState:"clean", BuildDate:"2019-12-22T23:14:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

-- soi
cert-manager
https
istio
kibana
kubernetes-operator

1 Answer

1/8/2020

I resolved the problem I had to rename the http port

apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
  name: monitoring
  namespace: {{ .Values.kibana.namespace }}  
spec:
  version: 7.4.2
  count: 1
  elasticsearchRef:
    name: monitoring
  http:
    tls:
      selfSignedCertificate:
        disabled: true
    service:
      spec:
        ports:
          - name: http
            port: 5601
            targetPort: 5601          
  podTemplate:
    spec:
      containers:
      - name: kibana
        env:
          - name: SERVER_BASEPATH
            value: ""      
        resources:
          requests:
            memory: 1Gi
            cpu: 0.5
          limits:
            memory: 2Gi
            cpu: 2 
    metadata:
      annotations:
        sidecar.istio.io/rewriteAppHTTPProbers: "true"
-- soi
Source: StackOverflow