We are migrating a legacy system to istio service mesh in kubernetes that uses consul DNS. Services within the mesh have to be addressed in the format of e.g. https://app.service.consul. Running this request within Istio service mesh has some challenges. While we managed to make it work with service entries and a custom envoy filter, we are now stuck making it work in a istio multi-cluster setup when calling the service in the external cluster. Below is a simplified diagram of how the service is called for reference.

Request from client app serivce when trying to reach the external cluster service fails:

curl https://static-server-https.service.consul
OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to static-server-https.service.consul:443


# but should be
hello world from clsuter-b

Verbose output:

curl https://static-server-https.service.consul
*   Trying 240.240.0.1:443...
* Connected to static-server-https.service.consul (240.240.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/certs/ca/ca-bundle.pem
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to static-server-https.service.consul:443 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to static-server-https.service.consul:443

Istio proxy outbound log to external cluster:

[2022-01-25T10:09:59.177Z] "- - -" 0 UF,URX - - "-" 0 0 10000 - "-" "-" "-" "-" "35.157.89.251:443" outbound|443||static-server-https.service.consul - 240.240.0.2:443 100.96.7.214:47428 - -

Request from client app serivce when trying to reach the internal cluster service succeeds:

curl https://static-server-https.service.consul
hello world from cluster-a

Istio proxy outbound log to internal cluster:

[2022-01-25T09:45:54.381Z] "- - -" 0 - - - "-" 102 191 8 - "-" "-" "-" "-" "100.70.235.145:443" outbound|443||static-server-https.service.consul 100.96.7.214:48320 240.240.0.2:443 100.96.7.214:55200 - -

Notes on istio proxy outbound logs

When comparing the istio proxy outbound logs, they look quite different from when using istio’s out of the box multi cluster communication. The following is an example log from the standard configuration, when using http://static-server-http.default.svc.cluster.local:8080 with response code 200:

[2022-01-25T10:02:40.334Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 29 7 6 "-" "curl/7.81.0-DEV" "7a46868c-7d22-9f79-89e5-b4b04835bb35" "static-server-http:8080" "18.195.240.125:15443" outbound|8080||static-server-http.default.svc.cluster.local 100.96.7.214:35468 100.96.7.223:8080 100.96.7.214:44864 - default

Compared to the log with response code 200 when using https://static-server-https.service.consul:

[2022-01-25T09:45:54.381Z] "- - -" 0 - - - "-" 102 191 8 - "-" "-" "-" "-" "100.70.235.145:443" outbound|443||static-server-https.service.consul 100.96.7.214:48320 240.240.0.2:443 100.96.7.214:55200 - -

In my personal view, this might be due to the custom envoy filter, but not sure at this point how we should interpret the difference as of now.

Notes on kiali service representation We also found that kiali renders the graph of the https configured connection with tcp instead of http. It is not known as of now why this is the case and if it is relevant.

Notes on the setup The setup is based on the multi-primary setup from the official documentation. I also have to note here that our setup is based on AWS infrastructure, which requires to supply the network loadbalancer ip addresses instead of its host names.

The east-est gateway for cluster communication is modified to allow traffic over the consul hostnames:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: cross-network-gateway
  namespace: istio-system
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        number: 15443
        name: tls
        protocol: TLS
      tls:
        mode: AUTO_PASSTHROUGH
      hosts:
        - "*.local"
        - "*.service.consul"

The target service is deployed with a service entry that configures the AWS NLB IP addresses as endoints. We also have a custom envoy filter to allow the use of https in our requests.

apiVersion: v1
kind: Service
metadata:
  name: static-server-https
  namespace: default
spec:
  selector:
    app: static-server-https
  ports:
    - name: https
      protocol: TCP 
      port: 443 
      targetPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-server-https
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: static-server-https
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: static-server-https
  template:
    metadata:
      name: static-server-https
      labels:
        app: static-server-https
      annotations:
        # the certificates for the custom envoy https filter
        sidecar.istio.io/userVolumeMount: '[{"name":"consul-external-cert", "mountPath":"/etc/certs/consul-external", "readonly":true},{"name":"root-ca", "mountPath":"/etc/certs/ca", "readonly":true}]'
        sidecar.istio.io/userVolume: '[{"name":"consul-external-cert", "secret":{"secretName":"consul-wildcard"}},{"name":"root-ca", "secret":{"secretName":"ca-bundle"}}]'
    spec:
      containers:
      - name: static-server-https
        image: hashicorp/http-echo:latest
        args:
          - -text="hello world from {{.Values.mesh.cluster.this.name}}"
          - -listen=:8080
        ports:
          - containerPort: 8080
            name: http
      serviceAccountName: static-server-https
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: static-server-https
  namespace: default
spec:
  hosts:
  - static-server-https.service.consul
  location: MESH_INTERNAL
  ports:
  - number: 443
    name: https
    protocol: TLS
  resolution: DNS
  endpoints: 
  # the AWS NLB addresses to reach the app in the external cluster
  - address: {{.Values.mesh.cluster.external.gateway.address1}}
    ports: 
      http: 15443 
  - address: {{.Values.mesh.cluster.external.gateway.address2}}
    ports: 
      http: 15443 
  - address: {{.Values.mesh.cluster.external.gateway.address3}}
    ports: 
      http: 15443 
  # the app endpoint for the local cluster
  - address: static-server-https.default
  subjectAltNames:
  - "spiffe://cluster.local/ns/default/sa/static-server-https"
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: static-server-https
spec:
  host: static-server-https.service.consul
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: custom-envoy-filter
spec:
  configPatches:
    - applyTo: FILTER_CHAIN
      match:
        context: SIDECAR_OUTBOUND
        listener:
          portNumber: 443
          filterChain:
            filter:
              name: "*.service.consul"
      patch:
        operation: MERGE
        value:
          transport_socket:
            name: tls
            typed_config:
              "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext"
              common_tls_context:
                tls_certificates:
                  - certificate_chain:
                      filename: /etc/certs/consul-external/tls.crt
                    private_key:
                      filename: /etc/certs/consul-external/tls.key
                validation_context:
                  trusted_ca:
                    filename: /etc/certs/ca/ca.crt