I'm trying to setup istio (v1.7.3) on AKS (v1.16.13) in a way that for some of the HTTP destinations a TLS Origination will be performed. So when one of my pods is invoking abc.mydomain.com with HTTP, the Egress request will be upgraded to HTTPS and the TLS verification done through the Egress gateway.

I have followed these 2 tutorials to achieve that:

• https://istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway-tls-origination-sds/
• https://istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway-tls-origination/

I ended up with something like this (abc.mydomain.com is an external URL so that why I created a ServiceEntry for it):

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: abc.mydomain.com
spec:
  hosts:
  - abc.mydomain.com
  ports:
  - number: 80
    name: http
    protocol: HTTP
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway
  namespace: istio-system
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - abc.mydomain.com
    tls:
      mode: ISTIO_MUTUAL
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-abc
  namespace: istio-system
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
  - name: abc
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - port:
          number: 443
        tls:
          mode: ISTIO_MUTUAL
          sni: abc.mydomain.com
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-abc-through-egress-gateway
  namespace: istio-system
spec:
  hosts:
  - abc.mydomain.com
  gateways:
  - istio-egressgateway
  - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        subset: abc
        port:
          number: 443
      weight: 100
  - match:
    - gateways:
      - istio-egressgateway
      port: 443
    route:
    - destination:
        host: abc.mydomain.com
        port:
          number: 443
      weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: originate-tls-for-abc
  namespace: istio-system
spec:
  host: abc.mydomain.com
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 443
      tls:
        mode: SIMPLE
        credentialName: abc # this must match the secret created earlier without the "-cacert" suffix
        sni: abc.mydomain.com

I'm creating a secret for my CA root with: kubectl create secret generic abc-cacert --from-file=ca.crt=mydomainrootca.crt -n istio-system

I've used the same certificate for my java applications and I can successfully invoke HTTPS for the same url using JKS. It seems the certificate is loaded properly into egress (kubectl logs -f -l istio=egressgateway -n istio-system):

2020-10-06T20:00:36.611607Z     info    sds     resource:abc-cacert new connection
2020-10-06T20:00:36.612907Z     info    sds     Skipping waiting for gateway secret
2020-10-06T20:00:36.612994Z     info    cache   GenerateSecret abc-cacert
2020-10-06T20:00:36.613063Z     info    sds     resource:abc-cacert pushed root cert to proxy

When I invoke curl abc.mydomain.com from a pod running on my cluster I'm getting this error from egress gateway:

[2020-10-06T19:33:40.902Z] "GET / HTTP/1.1" 503 UF,URX "-" "TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED" 0 91 172 - "192.244.0.191" "curl/7.64.0" "b618b1e6-e543-4053-bf2f-8ae56664545f" "abc.mydomain.com" "192.223.24.254:443" outbound|443||abc.mydomain.com - 192.244.0.188:8443 192.244.0.191:41306 abc.mydomain.com -

Any idea what I might be doing wrong? I'm quite new to istio and I don't understand all of the need of DestinationRule/VirtualService so please bare with me.

UPDATE1

After putting the DestinationRules in the namespace where my pod is running, I'm getting the following:

curl abc.mydomain.com
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.17.10</center>
</body>
</html>

Here is the output of istioctl proxy-status:

NAME                                                            CDS        LDS        EDS        RDS        ISTIOD                      VERSION
istio-egressgateway-695dc4fc7c-p5p42.istio-system               SYNCED     SYNCED     SYNCED     SYNCED     istiod-5c6b7b5b8f-csggg     1.7.3
istio-ingressgateway-5689f7c67-j54m7.istio-system               SYNCED     SYNCED     SYNCED     SYNCED     istiod-5c6b7b5b8f-csggg     1.7.3
test-5bbfdb8f4b-hg7vf.test                                      SYNCED     SYNCED     SYNCED     SYNCED     istiod-5c6b7b5b8f-csggg     1.7.3