Putting kibana behind nginx-ingress fails with a HTTP error

10/25/2019

I have deployed kibana in a kubernetes environment. If I give that a LoadBalancer type Service, I could access it fine. However, when I try to access the same via a nginx-ingress it fails. The configuration that I use in my nginx ingress is:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: my-ingress
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: kibana
              servicePort: {{ .Values.kibanaPort }}
            path: /kibana

I have launched my kibana with the following setting:

    - name: SERVER_BASEPATH
      value: /kibana

and I am able to access the kibana fine via the LoadBalancer IP. However when I try to access via the Ingress, most of the calls go through fine except for a GET call to vendors.bundle.js where it fails almost consistently.

The log messages in the ingress during this call is as follows:

2019/10/25 07:31:48 [error] 430#430: *21284 upstream prematurely closed connection while sending to client, client: 10.142.0.84, server: _, request: "GET /kibana/bundles/vendors.bundle.js HTTP/2.0", upstream: "http://10.20.3.5:3000/kibana/bundles/vendors.bundle.js", host: "1.2.3.4", referrer: "https://1.2.3.4/kibana/app/kibana"
10.142.0.84 - [10.142.0.84] - - [25/Oct/2019:07:31:48 +0000] "GET /kibana/bundles/vendors.bundle.js HTTP/2.0" 200 1854133 "https://1.2.3.4/kibana/app/kibana" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" 47 13.512 [some-service] 10.20.3.5:3000 7607326 13.513 200 506f778b25471822e62fbda2e57ccd6b

I am not sure why I get the upstream prematurely closed connection while sending to client across different browsers. I have tried setting proxy-connect-timeout and proxy-read-timeout to 100 seconds and even then it fails. I am not sure if this is due to some kind of default size or chunks.

Also it is interesting to note that only some kibana calls are failing and not all are failing.

In the browser, I see the error message:

GET https://<ip>/kibana/bundles/vendors.bundle.js net::ERR_SPDY_PROTOCOL_ERROR 200

in the developer console.

Anyone has any idea on what kind of config options I need to pass to my nginx-ingress to make kibana proxy_pass fine ?

-- Sankar
google-cloud-platform
kibana
kubernetes
nginx
nginx-ingress

1 Answer

10/29/2019

I have found the cause of the error. The vendors.bundle.js file was relatively bigger and since I was accessing from a relatively slow network, the requests were terminated. The way I fixed this is, by adding to the nginx-ingress configuration the following fields:

nginx.ingress.kubernetes.io/proxy-body-size: 10m (Change this as you need)
nginx.ingress.kubernetes.io/proxy-connect-timeout: "100"
nginx.ingress.kubernetes.io/proxy-send-timeout: "100"
nginx.ingress.kubernetes.io/proxy-read-timeout: "100"
nginx.ingress.kubernetes.io/proxy-buffering: "on"
-- Sankar
Source: StackOverflow