We have an app that we are trying to move into Istio mesh. One of the services makes requests to metadata.google.internal in order to finish configuring the environment.

I created ServiceEntry type object to whitelist metadata.google.internal, as follows (have tried different combos of this):

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: google-metadata-server
spec:
  hosts:
  - metadata.google.internal
  location: MESH_EXTERNAL
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS

For some reason I am not being able to get the proper responses, so the service fails. When checking the logs on the istio-proxy container, in the service, this is the errors I get:

[2019-02-07T15:29:22.834Z] "GET /computeMetadata/v1/project/project-idHTTP/1.1" 200 - 0 14 2 1 "-" "Google-HTTP-Java-Client/1.27.0 (gzip)" "513f6e25-57ce-4cf0-a273-d391b3da604b" "metadata.google.internal" "169.254.169.254:80" outbound|80||metadata.google.internal - 169.254.169.254:80 10.16.0.29:58790
[2019-02-07T15:29:22.865Z] "GET /HTTP/1.1" 404 NR 0 0 0 - "-" "Google-HTTP-Java-Client/1.27.0 (gzip)" "490fe0dd-e03a-4460-a63b-7ed84a168d39" "169.254.169.254" "-" - - 169.254.169.254:80 10.16.0.29:58794
[2019-02-07T15:29:22.883Z] "GET /HTTP/1.1" 404 NR 0 0 0 - "-" "Google-HTTP-Java-Client/1.27.0 (gzip)" "dac0e762-d799-4457-ae48-75179e2f0b82" "169.254.169.254" "-" - - 169.254.169.254:80 10.16.0.29:58794
[2019-02-07T15:29:22.886Z] "GET /HTTP/1.1" 404 NR 0 0 0 - "-" "Google-HTTP-Java-Client/1.27.0 (gzip)" "3411a0be-6d29-42f3-b01a-567edf2cc3e2" "169.254.169.254" "-" - - 169.254.169.254:80 10.16.0.29:58794
[2019-02-07T15:29:47.781Z] "GET /computeMetadata/v1/project/project-idHTTP/1.1" 200 - 0 14 4 3 "-" "Google-HTTP-Java-Client/1.27.0 (gzip)" "7115bf46-e7e9-4b2f-ba37-10cd6b8c9dea" "metadata.google.internal" "169.254.169.254:80" outbound|80||metadata.google.internal - 169.254.169.254:80 10.16.0.29:58876

As you can see I am getting 200 for the requests to /computeMetadata/v1/project/project-id, but it fails for /. Not sure why.

With istio-sidecar-injector pod everything seems to be fine. Nothing to point out in the logs.

What I did is to create a simple deployment with Istio, in the same cluster, same namespace, and telnet the metadata server manually:

root@demo-675c758c68-vrbcv:/# telnet metadata.google.internal 80
Trying 169.254.169.254...
Connected to metadata.google.internal.
Escape character is '^]'.
GET / HTTP/1.1      
Host: metadata.google.internal

HTTP/1.1 200 OK
metadata-flavor: Google
content-type: application/text
date: Thu, 07 Feb 2019 16:35:52 GMT
server: envoy
content-length: 22
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
x-envoy-upstream-service-time: 1

0.1/
computeMetadata/

I am not understanding why am I getting these 404. Any idea what is happening?