Unable to access pod over HTTPS via Istio Gateway (running as ELB) on AWS EKS with mTLS enabled on the cluster

2/22/2019

We have installed Istio 1.0.5 on a newly launched EKS cluster via Helm.

helm install \
    --wait \
    --name istio \
    --namespace istio-system \
    --set tracing.enabled=true \
    --set kiali.enabled=true \
    --set grafana.enabled=true \
    --set global.mtls.enabled=true \
    --set servicegraph.enabled=true \
    --set global.proxy.includeIPRanges="10.10.0.0/16\,172.20.0.0/16" \
    install/kubernetes/helm/istio 

(the IP ranges are pod and service CIDR ranges of the cluster)

This is what we want to achieve: istio ingress gateway

To enable attaching ACM certs to the ELB launched by the istio-ingressgateway, following annotations were added to the istio-ingressgateway service:

service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:ACCOUNT:certificate/MY_CERT
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https

The Ingress Gateway Loadbalancer service is getting created as expected:

enter image description here enter image description here

We are following this link to implement the above scenario:

https://doc.istio.cn/en/docs/examples/advanced-gateways/ingress-sni-passthrough/

The certificates and secrets are created according to the above Istio documentation. The Service and Deployment resources are created as follows:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx
  labels:
    run: my-nginx
spec:
  ports:
  - port: 443
    protocol: TCP
  selector:
    run: my-nginx

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 1
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 443
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx
          readOnly: true
        - name: nginx-server-certs
          mountPath: /etc/nginx-server-certs
          readOnly: true
      volumes:
      - name: nginx-config
        configMap:
          name: nginx-configmap
      - name: nginx-server-certs
        secret:
          secretName: nginx-server-certs

The Gateway and VirtualService resources are created as follows:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: mygateway
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: PASSTHROUGH
    hosts:
    - nginx.example.com

---

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
spec:
  hosts:
  - nginx.example.com
  gateways:
  - mygateway
  tls:
  - match:
    - port: 443
      sni_hosts:
      - nginx.example.com
    route:
    - destination:
        host: my-nginx
        port:
          number: 443

The resources are getting created as expected:

enter image description here

We are able to get a response from the nginx container from within the pod:

curl --insecure --verbose https://localhost

enter image description here

We are also able to get a response from the nginx pod via its service from another pod:

curl --insecure --verbose https://my-nginx

enter image description here

However, when we try to access the nginx pod via the Istio ingress gateway, the service is not accessible:

curl --insecure --verbose https://a1a99cb2d34f711e9865b0295f80a9c0-303710594.us-east-1.elb.amazonaws.com

enter image description here

After enabling access log for the ELB, this is the relevant log line:

2019-02-22T12:36:08.960931Z a1a99cb2d34f711e9865b0295f80a9c0 3.84.67.219:56786 - -1 -1 -1 503 0 0 0 "GET https://a1a99cb2d34f711e9865b0295f80a9c0-303710594.us-east-1.elb.amazonaws.com:443/ HTTP/1.1" "curl/7.55.1" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2

We got inside the istio-ingressgateway pod and tried accessing the nginx pod via its service name(my-nginx), service ip(172.20.198.12), service dns(my-nginx.default.svc.cluster.local), and pod ip(10.10.157.58). But none of them were accessible:

enter image description here

Do we need to create any extra VirtualService or ServiceEntry? When we tried accessing a Deployment exposed on port 80, via the Istio ingressgateway, we were able to successfully hit the pod. The issue arises only when we try to enable HTTPS on the Istio ingressgateway service. Is this occuring because we have enabled mTLS? Please help us find any pointers as to how to proceed.

-- Akshay Elavia
amazon-elb
amazon-web-services
istio
kubernetes
kubernetes-ingress

0 Answers