K
Q

Question

Istio Egress MTLs for external services

1/8/2020

I'm currently (and unsuccessfully) trying to setup MTLs via istio-egressgateway to access an external K8s cluster service. I'm following the intructions specified on istio docs but nothing works as expected, and I'm not able to see where I'm wrong.

Environment

3 VMs under VMWare ESXi (1 master, 2 Nodes)
OS : CentOS 7 with kernel 5.2.10-1.el7 from elrepo
k8s networking : cilium -1.6.4
Certificates : cert-manager-0.12 nowebhook
- ClusterIssuer CA cert = Imported from private PKI
K8s external load balancer : metallb-0.8.3

kubectl and istio used in K8s cluster

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.4", GitCommit:"224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba", GitTreeState:"clean", BuildDate:"2019-12-11T12:37:43Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

$ istioctl version
client version: 1.2.10
citadel version: 1.2.10
egressgateway version: 1.2.10
galley version: 1.2.10
ingressgateway version: 1.2.10
pilot version: 1.2.10
policy version: 1.2.10
sidecar-injector version: 1.2.10
telemetry version: 1.2.10

istio-egressgateway has been redeployed as indicated in istio docs

This are the applied .yaml is almost a copy-paste from the example, with a minor modification to point to an external service via it's IP address

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: nginx
  namespace: tools-istio
spec:
  hosts:
  - <my.hostname>
  ports:
  - number: 80
    name: http
    protocol: HTTP
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
  endpoints:
  - address: <private_ip_address>
    ports:
      https: 443
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
  namespace: tools-istio
spec:
  hosts:
  - <my.hostname>
  tls:
  - match:
    - port: 443
      sni_hosts:
      - <my.hostname>
    route:
    - destination:
        host: <my.hostname>
        port:
          number: 443
      weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway
  namespace: tools-istio
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - <my.hostname>
    tls:
      mode: MUTUAL
      serverCertificate: /etc/certs/cert-chain.pem
      privateKey: /etc/certs/key.pem
      caCertificates: /etc/certs/root-cert.pem
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-nginx
  namespace: tools-istio
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
  - name: nginx
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - port:
          number: 443
        tls:
          mode: ISTIO_MUTUAL
          sni: <my.hostname>
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-nginx-through-egress-gateway
  namespace: tools-istio
spec:
  hosts:
  - <my.hostname>
  gateways:
  - istio-egressgateway
  - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        subset: nginx
        port:
          number: 443
      weight: 100
  - match:
    - gateways:
      - istio-egressgateway
      port: 443
    route:
    - destination:
        host: <my.hostname>
        port:
          number: 443
      weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: originate-mtls-for-nginx
  namespace: tools-istio
spec:
  host: <my.hostname>
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 443
      tls:
        mode: ISTIO_MUTUAL
        sni: <my.hostname>
        # I've also tried with
        # mode: MUTUAL
        # clientCertificate: /etc/nginx-external-tls/tls.crt
        # privateKey: /etc/nginx-external-tls/tls.key
        # caCertificates: /etc/nginx-external-tls/ca.crt

When making a query to the external service via curl we should get the HTML web code, but instead, it tries to execute the query with HTTP, not HTTPS, why ??. istio-proxy from debugger pod should translate HTTP into HTTPS using Citadel certificates

 $ kubectl -n tools-istio exec -it debugger-5844b6d674-4fvt9 -c debugger -- curl http://<my.hostname> -v
*   Trying <my.hostname>:80...
* TCP_NODELAY set
* Connected to <my.hostname> (<private_ip_address>) port 80 (#0)
> GET / HTTP/1.1
> Host: <my.hostname>
> User-Agent: curl/7.66.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< content-length: 91
< content-type: text/plain
< date: Wed, 08 Jan 2020 09:29:09 GMT
< server: envoy
<
* Connection #0 to host <my.hostname> intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure

When verifying proxy-status I get the following

$ istioctl proxy-status
NAME                                                   CDS                            LDS        EDS               RDS          PILOT                            VERSION
debugger-5844b6d674-4fvt9.tools-istio                  STALE (Never Acknowledged)     SYNCED     SYNCED (51%)      SYNCED       istio-pilot-5f45f6768c-fmvr7     1.2.10
istio-egressgateway-5ff889c5fd-jtz55.istio-system      SYNCED                         SYNCED     SYNCED (100%)     SYNCED       istio-pilot-5f45f6768c-fmvr7     1.2.10
istio-ingressgateway-779cfdd879-bb7w7.istio-system     STALE (Never Acknowledged)     SYNCED     SYNCED (96%)      NOT SENT     istio-pilot-5f45f6768c-fmvr7     1.2.10

Which according with official istio docs can mean, either a missconfiguration in .yaml, either an istio bug

When checking endpoint everything looks fine

$ istioctl proxy-config endpoint debugger-5844b6d674-4fvt9.tools-istio | grep <my.hostname>
<private_ip_address>:80         HEALTHY     outbound|80||<my.hostname>
<private_ip_address>:443        HEALTHY     outbound|443||<my.hostname>

Othewise debugger pod istio-proxy logs show the errors below. But the external certificate is not at pod level, but in a recompiled istio-egressgateway (again this is nothing funcy, I've just followed istio docs), so why istio-proxy is looking for the external certificates ??

[2020-01-08 07:56:04.827][17][debug][init] [external/envoy/source/common/init/watcher_impl.cc:27] init manager Cluster outbound|443||afv.test.recouv destroyed
[2020-01-08 07:56:04.827][17][warning][config] [external/envoy/source/common/config/grpc_mux_subscription_impl.cc:73] gRPC config for type.googleapis.com/envoy.api.v2.Cluster rejected: Error adding/updating cluster(s) outbound|443||<my.hostname>: Invalid path: /etc/nginx-external-tls/tls.crt

Pilot logs shows the same, istio-proxy container of debugger pod keep looking for external service CA certificate, but this certs are in egressgateway

$ kubectl -n istio-system logs istio-pilot-5f45f6768c-fmvr7 -c discovery | grep <my.hostname>

2020-01-08T08:48:54.536073Z     warn    ads     ADS:CDS: ACK ERROR 127.0.0.1:60012 sidecar~10.27.187.219~debugger-5844b6d674-4fvt9.tools-istio~tools-istio.svc.cluster.local-15433 (debugger-5844b6d674-4fvt9.tools-istio) version_info:"2020-01-08T07:50:00Z/110" node:<id:"sidecar~10.27.187.219~debugger-5844b6d674-4fvt9.tools-istio~tools-istio.svc.cluster.local" cluster:"debugger.tools-istio" metadata:<fields:<key:"CONFIG_NAMESPACE" value:<string_value:"tools-istio" > > fields:<key:"INCLUDE_INBOUND_PORTS" value:<string_value:"8080" > > fields:<key:"INTERCEPTION_MODE" value:<string_value:"REDIRECT" > > fields:<key:"ISTIO_META_INSTANCE_IPS" value:<string_value:"10.27.187.219,10.27.187.219,fe80::a07f:d0ff:feec:5e71" > > fields:<key:"ISTIO_PROXY_SHA" value:<string_value:"istio-proxy:f9c0feb10cc42277e97dd080a2e045f62d1739a9" > > fields:<key:"ISTIO_PROXY_VERSION" value:<string_value:"1.1.3" > > fields:<key:"ISTIO_VERSION" value:<string_value:"1.2.10" > > fields:<key:"POD_NAME" value:<string_value:"debugger-5844b6d674-4fvt9" > > fields:<key:"app" value:<string_value:"debugger" > > fields:<key:"istio" value:<string_value:"sidecar" > > fields:<key:"pod-template-hash" value:<string_value:"5844b6d674" > > fields:<key:"prometheus.io/path" value:<string_value:"/metrics" > > fields:<key:"prometheus.io/port" value:<string_value:"8080" > > fields:<key:"prometheus.io/scrape" value:<string_value:"true" > > fields:<key:"sidecar.istio.io/rewriteAppHTTPProbers" value:<string_value:"true" > > > locality:<> build_version:"f9c0feb10cc42277e97dd080a2e045f62d1739a9/1.11.0-dev/Modified/RELEASE/BoringSSL" > type_url:"type.googleapis.com/envoy.api.v2.Cluster" response_nonce:"59a1bb9a-aca6-47c0-b718-a86d7e392db0" error_detail:<code:13 message:"Error adding/updating cluster(s) outbound|443||afv.test.recouv: Invalid path: /etc/nginx-external-tls/tls.crt" >
  message: "Error adding/updating cluster(s) outbound|443||<my.hostname>: Invalid path: /etc/nginx-external-tls/tls.crt"

In istio-egressgateway everything seams to be OK

$ kubectl -n istio-system logs istio-egressgateway-5ff889c5fd-jtz55
[2020-01-08 08:59:53.737][24][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:527] updating TLS cluster outbound_.80_._.<my.hostname>
[2020-01-08 08:59:53.737][24][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:975] membership update for TLS cluster outbound_.80_._.<my.hostname> added 1 removed 0
[2020-01-08 08:59:53.737][24][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:527] updating TLS cluster outbound_.443_._.<my.hostname>
[2020-01-08 08:59:53.737][24][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:975] membership update for TLS cluster outbound_.443_._.<my.hostname> added 1 removed 0

And of course, certificates are present as they should in istio-egressgateway. I've used a cert-manager cluster issuer to deliver a certificate for the external service, like that the AC and it's secrets are already on the cluster

# External AC
$ kubectl -n istio-system exec -it istio-egressgateway-5ff889c5fd-jtz55 -- ls /etc/cluster-issuer-tls
tls.crt  tls.key

# Client
$ kubectl -n istio-system exec -it istio-egressgateway-5ff889c5fd-jtz55 -- ls /etc/nginx-external-tls
ca.crt  tls.crt  tls.key

Has anyone any clue about what I'm missing ?? Is this a missconfiguration or a bug ??

Any help will be really appreciated, I've been stucked with this since before Christmas holidays

Cheers

-- abeleaveicanfly but I can't

istio

kubernetes

mtls

K
Q

Istio Egress MTLs for external services

Similar Questions

0 Answers

KQ

Istio Egress MTLs for external services

Similar Questions

0 Answers

K
Q