I've been trying to setup an externally facing GRPC payments microservice client with automatic cert renewal with tls.

So far I've set up the certmanager with the certificate renewal correctly however it appears my gateway is not forwarding traffic correctly as kubectl -n istio-system describe challenge payments-cert shows the challenge is erroring out due to HTTP 404 being returned.

I used grpcurl to test the liveliness of the service, this was the result:

Failed to dial target host "payments.mywebsite.com:443": read tcp 192.168.0.16:58849 ->xx.xx.xxx.xx:443: read: connection reset by peer

Which indicates the ip has been registered by the dns correctly, and the address is indeed arriving on 443, so there must be an issue with my Gateway -> VirtualService -> Service -> Deployment setup.

When describing the istio ingress (kubectl get svc -n istio-system istio-ingressgateway) I get:

istio-ingressgateway LoadBalancer 10.0.13.184 xx.xx.xx.xx ...443:31390/TCP...

Which seems like its not forwarding 443 to the intended ports?

I want to have traffic coming in on 'payments.mywebsite.com:443' (with automatic certificate renewal with let-encrypt) to get routed to my grpc backend on container port 50051. I have been having a very hard time deciphering the Istio docs and resources to get this done!

Any help will be warmly welcomed!

Gateway

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: payments-gateway
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - payments.mywebsite.com
    port:
      name: payments-gateway-https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: payments-cert
      mode: SIMPLE
      privateKey: sds
      serverCertificate: sds

Virtual Service

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: payments-virtual
spec:
  hosts:
  - "payments.mywebsite.com"
  gateways:
  - payments-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        port:
          number: 50051
        host: payments-service

Service

apiVersion: v1
kind: Service
metadata:
  name: payments-service
  labels:
    app: payments-service
spec:
  ports:
  - port: 50051
    name: grpc
  selector:
    app: payments-server

Deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: payments-server
  name: payments-server
spec:
  selector:
    matchLabels:
      app: payments-server
  replicas: 1
  strategy: {}
  template:
    metadata:
      labels:
        app: payments-server
        version: v1
    spec:
      containers:
      - image: gcr.io/mywebsite/payments_server:v1
        name: payments-server
        ports:
        - containerPort: 50051
        resources: {}
      restartPolicy: Always
status: {}

Edit 1: Added logs from gateway

2020-01-21T14:31:48.841505Z     error   k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:98: Failed to list *v1.Secret: Get https://10.0.0.1:443/api/v1/namespaces/istio-system/secrets?limit=500&resourceVersion=0: net/http: TLS handshake timeout
2020-01-21T14:32:13.173045Z     info    secretFetcherLog        scrtUpdated is called on kubernetes secret payments-cert
2020-01-21T14:32:13.173124Z     warn    secretFetcherLog        failed load server cert/key pair from secret payments-cert: server cert or private key is empty
2020-01-21T14:32:13.173131Z     warn    secretFetcherLog        failed load server cert/key pair from secret payments-cert: server cert or private key is empty
2020-01-21T14:32:13.173137Z     info    secretFetcherLog        secret payments-cert does not change, skip update
2020-01-21T14:32:13.173257Z     info    Trace[1103410]: "Reflector k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:98 ListAndWatch" (started: 2020-01-21 14:31:49.841728972 +0000 UTC m=+44333.359224094) (total time: 23.331269811s):`

Edit 2: Added proxy sidecar logs (bearer token omitted)

2020-01-21T14:47:51.784617Z     info    curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: sidecar-injector/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer ***" 'https://10.0.0.1:443/apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations?fieldSelector=metadata.name%3Distio-sidecar-injector&resourceVersion=299045&timeoutSeconds=449&watch=true'
2020-01-21T14:47:51.787107Z     info    GET https://10.0.0.1:443/apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations?fieldSelector=metadata.name%3Distio-sidecar-injector&resourceVersion=299045&timeoutSeconds=449&watch=true 200 OK in 2 milliseconds
2020-01-21T14:47:51.787126Z     info    Response Headers:
2020-01-21T14:47:51.787132Z     info        Audit-Id: 21862701-0185-4e25-b26e-ff78ddb6de1e
2020-01-21T14:47:51.787136Z     info        Content-Type: application/json
2020-01-21T14:47:51.787140Z     info        Date: Tue, 21 Jan 2020 14:47:51 GMT

Edit 3: Payments cert structure and Cluster issuer

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: payments-cert
  namespace: istio-system
spec:
  commonName: payments.mywebsite.com
  dnsNames:
  - payments.mywebsite.com
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-prod
  secretName: payments-cert

---

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
 labels:
   name: letsencrypt-prod
 name: letsencrypt-prod
 namespace: istio-system
spec:
 acme:
   email: mywebsite@gmail.com
   solvers:
    - http01:
        ingress:
          class:  istioIngress
   privateKeySecretRef:
     name: letsencrypt-staging
   server: https://acme-v02.api.letsencrypt.org/directory

Edit 4: output of describe payments-cert (+ additional logs from istio sidecar injector)

Name:         payments-cert
Namespace:    istio-system
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: payments-cert
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: letsencrypt-prod

Type:  kubernetes.io/tls

Data
====
tls.crt:  0 bytes
tls.key:  1679 bytes
ca.crt:   0 bytes

I also took a look at the emitted events from the istio sidecar injector on google cloud, these logs are recorded:

(Message): No matching pods found   
(Reason): NoPods    
(First Seen): Jan 21, 2020, 2:32:47 PM  
(Last Seen): Jan 21, 2020, 3:28:17 PM   
(counts): 112