Good Morning. I have a GRPC server that I want to serve on Google Kubernetes Engine. My cluster already has the nginx-ingress controller installed, and I'm currently using this to serve http/https traffic. This is the ingress resource I've tried to make to host the GRPC server:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: grpc-ingress-nginx
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/grpc-backend: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  namespace: default
spec:
  tls:
  - hosts:
    - bar.foo.com
    secretName: reploy-tls
  rules:
  - host: bar.foo.com
    http:
      paths:
      - backend:                                                                                                                                                                                                      
          serviceName: bar-backend-service
          servicePort: 50051

And here's the service/deployment for the app:

apiVersion: v1
kind: Service
metadata:
  name: bar-backend-service
  namespace: default
spec:
  selector:
    app: bar-backend-app
  ports:
  - port: 50051
    targetPort: 50051
    name: grpc
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: bar-backend
  labels:
    app: bar-backend-app
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: bar-backend-app
    spec:
      containers:
      - name: bar-backend-image
        image: gcr.io/himank-jay/bar-backend-image:TAG
        ports:
        - containerPort: 50051
          name: grpc

When I run grpc_cli ls bar.foo.com:443 (using grpc_cli), I get the following error:

{"created":"@1580833741.460274000","description":"Error received from peer ipv4:x.x.x.x:x","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Socket closed","grpc_status":14}

And the error from the nginx-controller is as follows:

x.x.x.x - - [04/Feb/2020:16:28:46 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.020 [] [] - - - - xxxxx

Any ideas on what's wrong here? Or any thoughts on how to debug this?