K
Q

Kubernetes - Ingress with gRPC

7/28/2020

I have a very simple grpc-server. When I expose them directly via NodePort everything works just fine. I do not have any issues with this.

grpc-client --> NodePort --> grpc-server

I changed the NodePort service to ClusterIP service and tried to use ingress controller to route the traffic to the grpc-server. The set up is like this. ingress and grpc-server are part of k8s cluster. grpc-client is outside - which is my local machine.

grpc-client --> ingress --> clusterip --> grpc-server

My ingress is like this.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: web-ingress
  namespace: test
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;
      grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster.local:$service_port;    
spec:
  rules:
  - http:
      paths:
      - backend:
         serviceName: grpc-server
         servicePort: 6565

Whenever I send a request, I get en exception like this in the client side.

io.grpc.StatusRuntimeException: INTERNAL: HTTP/2 error code: FRAME_SIZE_ERROR
Received Rst Stream

Ingress log is like this

127.0.0.1 - - [28/Jul/2020:03:52:38 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - c9d46af55aa8a2b450f9f72e9d550023
127.0.0.1 - - [28/Jul/2020:03:52:38 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.000 [] [] - - - - a9e0f02851473467fc553f2a4645377d
127.0.0.1 - - [28/Jul/2020:03:52:40 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - fcc0668f300bd5e7e12325ae7c26fa9d
127.0.0.1 - - [28/Jul/2020:03:52:41 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - dea2b8eb8f6884ac512654af3c2e6b47
127.0.0.1 - - [28/Jul/2020:03:52:43 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - 0357acafce928ee05aaef1334eed1b69
127.0.0.1 - - [28/Jul/2020:03:52:44 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - df9b75d79d9177e39a901dfa020912f2
127.0.0.1 - - [28/Jul/2020:03:52:44 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.000 [] [] - - - - bb27acd5b86fd057af1ca8fe93f7d857
127.0.0.1 - - [28/Jul/2020:03:52:44 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - 2ced9207652fc564bb94e0d4a14ae6fc
127.0.0.1 - - [28/Jul/2020:03:52:45 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - b99fab13df2f64a6a0a901237dadfa27
127.0.0.1 - - [28/Jul/2020:03:52:45 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.002 [] [] - - - - 40cb75f4e4b3df28be66044a0825f1c6
127.0.0.1 - - [28/Jul/2020:03:52:46 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - 89cac6709278b4e54b0349075500bf74
127.0.0.1 - - [28/Jul/2020:03:52:46 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.000 [] [] - - - - 0b032199a053069df185c620de497c90
127.0.0.1 - - [28/Jul/2020:03:52:46 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.001 [] [] - - - - 5a07b1e24ea3e8e2066a7ed8950f7040
127.0.0.1 - - [28/Jul/2020:03:53:46 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" 0 0.000 [] [] - - - - 0f63eec0d0ccb9cc87c2fbb1fdfae48f

Looks like the grpc-server never received any request as I do not see any log there.

What is going on?

-- RamPrakash
grpc
grpc-java
kubernetes
kubernetes-ingress
nginx

Similar Questions

What the status of pods should be if the health check doesn't meet successThreshold and failureThreshold, neither
How to migrate Unicorn Rails base application to Kubernetes
Kubernetes Cronjob triggers Job immediately even though the time exceeded
Python code that works locally does not work on Kubernetes clusters
Best practices for organizing kubernetes yaml when dealing with multiple environment in a project

1 Answer

7/31/2020

That error you encountered comes from the default configuration of the nginx ingress controller (this is taken from the ingress github config): 

// Enables or disables the HTTP/2 support in secure connections
// http://nginx.org/en/docs/http/ngx_http_v2_module.html
// Default: true
UseHTTP2 bool `json:"use-http2,omitempty"`

This can be also verified by checking your ingress controller settings : 

kubectl exec -it -n <name_space> <nginx_ingress_controller_pod> -- cat /etc/nginx/nginx.conf 

## start server _
   server {
	      server_name _ ;

	      listen 80 default_server reuseport backlog=511 ;
	      listen 443 default_server reuseport backlog=511 ssl http2 ;

You're trying to connect through port 80 for which there is no http2 support in default configuration. GRCP works only with http2 so you should try to make a request using port 443.

Let me know if it works.

-- acid_fuji
Source: StackOverflow