Getting 404 on all outbound HTTP calls from pods inside istio mesh

5/15/2019

I have istio v1.1.6 installed on Kubernetes v1.11 using Helm chart provided in istio repository with bunch of overrides including:

global:
  outboundTrafficPolicy:
    mode: ALLOW_ANY
pilot:
  env:
    PILOT_ENABLE_FALLTHROUGH_ROUTE: "1"
mixer:
  enabled: true
galley:
  enabled: true
security:
  enabled: false

The problem is that I can't make any simple outbound HTTP request to a service running on port 80 (inside & outside of the mesh) from a pod which is inside istio mesh and has istio-proxy injected as sidecar. The response is always 404:

user@pod-12345-12345$ curl -v http://httpbin.org/headers
* Hostname was NOT found in DNS cache
*   Trying 52.200.83.146...
* Connected to httpbin.org (52.200.83.146) port 80 (#0)
> GET /headers HTTP/1.1
> User-Agent: curl/7.38.0
> Host: httpbin.org
> Accept: */*
>
< HTTP/1.1 404 Not Found
< date: Wed, 15 May 2019 05:43:24 GMT
* Server envoy is not blacklisted
< server: envoy
< content-length: 0
<
* Connection #0 to host httpbin.org left intact

The response flag in the istio-proxy logs from envoy states that it can't find the proper route:

"GET / HTTP/1.1" 404 NR "-" 0 0 0 - "-" "curl/7.38.0" "238d0799-f83d-4e5e-94e7-79de4d14fa53" "httpbin.org" "-" - - 172.217.27.14:80 100.99.149.201:52892 -

NR: No route configured for a given request in addition to 404 response code.

Probably worths to add that:

  • Other outbound calls to any ports other than 80 works perfectly fine.
  • Checking proxy-status also shows nothing: all pods are SYNCED.
  • mTLS is disabled
  • The example above is a call to external service but calls to internal services (eg: curl another-service.svc.cluster.local/health) have the same issue.
  • I expecte calls to internal mesh services work out of the box, tho even I tried to define DestinationRoute and ServiceEntry and that did not help as well.
  • I don't really want to add traffic.sidecar.istio.io/excludeOutboundIPRanges: "0.0.0.0/0" annotation to deployment as according to docs:

    this approach completely bypasses the sidecar, essentially disabling all of Istio’s features for the specified IPs

Any idea where else I can look or what is missing?

-- Rahman
istio
kubernetes

1 Answer

5/29/2019

To me it looks like you have a short connection timeouts defined in your istio-proxy sidecar, please check here similar github issue of Envoy's project.

btw. as @Robert Panzer mentioned, sharing the whole dump of istio-proxy config would help much in investigating your particular case.

-- Nepomucen
Source: StackOverflow