K
Q

Question

Using istio-proxy with nginx pod errors out with 400 Request Header Or Cookie Too Large on kubernetes

12/5/2018

I am trying to use nginx with istio sidecar-injection enabled in the namespace on my kubernetes cluster.

This means that the pod is running with a composition of [istio-proxy, istio-init and nginx] containers.

When I send a request (curl) to the nginx service at <svc-name>.default.svc.cluster.local:80/version from a in-mesh pod (sleep pod), where the expected behavior is to hit the redirect/proxy_pass rule for /version should proxy it to the upstream service, I get a status code 400 with the following contents :

root@sleep-86f6b99f94-bqmbv:/# curl -v http://istiotesthelm-sentry.default.svc.cluster.local/version
* Hostname was NOT found in DNS cache
*   Trying 100.64.143.31...
* Connected to istiotesthelm-sentry.default.svc.cluster.local (100.64.143.31) port 80 (#0)
> GET /version HTTP/1.1
> User-Agent: curl/7.35.0
> Host: istiotesthelm-sentry.default.svc.cluster.local
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< date: Wed, 05 Dec 2018 01:45:24 GMT
< content-type: text/html
< content-length: 242
* Server envoy is not blacklisted
< server: envoy
< x-envoy-upstream-service-time: 1436
<
<html>
<head><title>400 Request Header Or Cookie Too Large</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>Request Header Or Cookie Too Large</center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host istiotesthelm-sentry.default.svc.cluster.local left intact

Looking at the nginx logs and/or the istio-proxy logs I see a slew of message which seems like nginx/envoy is trampling over itself -

Envoy (istio-proxy) logs

[2018-12-05T01:45:23.769Z] "GET /version HTTP/1.1" 400 - 0 242 877 877 "127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1" "curl/7.35.0" "8c645f63-cc1c-9f1b-9c65-5b51eae6aa08" "istiotesthelm-sentry.default.svc.cluster.local" "100.96.19.77:8081"

Nginx Logs

[05/Dec/2018:01:45:24 +0000] rlm="" "1.0" "off" "" "/version" "GET" "" "0" "" "" "" "" "" "400" "1.361" "437" "127.0.0.1" "-" "-" "HTTP/1.1" "http" "8c645f63-cc1c-9f1b-9c65-5b51eae6aa08" "242" "710" "istiotesthelm-sentry.default.svc.cluster.local" "-" "curl/7.35.0" "-" "127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1" "-" "1311560" "disabled" "1.359" "0.000" "1.359" "200" ""
[05/Dec/2018:01:45:24 +0000] rlm="" "1.0" "off" "" "/version" "GET" "" "0" "" "" "" "" "" "400" "1.364" "437" "127.0.0.1" "-" "-" "HTTP/1.1" "http" "8c645f63-cc1c-9f1b-9c65-5b51eae6aa08" "242" "699" "istiotesthelm-sentry.default.svc.cluster.local" "-" "curl/7.35.0" "-" "127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1" "-" "369741" "disabled" "1.363" "0.000" "1.363" "200" ""
[05/Dec/2018:01:45:24 +0000] rlm="" "1.0" "off" "" "/version" "GET" "" "0" "" "" "" "" "" "400" "1.366" "437" "127.0.0.1" "-" "-" "HTTP/1.1" "http" "8c645f63-cc1c-9f1b-9c65-5b51eae6aa08" "242" "688" "istiotesthelm-sentry.default.svc.cluster.local" "-" "curl/7.35.0" "-" "127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1, 127.0.0.1" "-" "422420" "disabled" "1.365" "0.000" "1.365" "200" ""

Would really appreciate if someone could help me understand, what is happening in this situation.

-- nitimalh

envoyproxy

istio

kubernetes

nginx

K
Q

Using istio-proxy with nginx pod errors out with 400 Request Header Or Cookie Too Large on kubernetes

Similar Questions

0 Answers

KQ

Using istio-proxy with nginx pod errors out with 400 Request Header Or Cookie Too Large on kubernetes

Similar Questions

0 Answers

K
Q