I install Istio on GKE and run the application.
Although there is no problem when accessing with curl, Ingressgateway returns a status code different from the status code of Pod's proxy by some image request when accessing from the browser. Specifically, 200 and 302 etc. are returned as 500 or 504. Resources to become 500 or 504 differ every time, but it is 1 or 2 out of about 100 image requests. And if you request another 500 or 504 request again, the correct response will come back without problems.
Do you know what is causing this kind of reason?
The environment is like this.
GKE 1.10.11-gke.1
Istio 1.0.4
helm install install/kubernetes/helm/istio --name istio --namespace istio-system --set tracing.enabled=true --set kiali.enabled=true --set global.proxy.includeIPRanges="10.0.0.0/8"
Below is the log obtained from Stackdriver Logging.
Ingressgateway log.
"[2019-01-22T09:16:17.048Z] \"GET /my/app/image.pngHTTP/2\" 504 UT 0 24 60001 - \"xxx.xxx.xxx.xxx\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36\" \"c0abe3be-1153-45c5-bd8e-067ab597feb4\" \"my.app.com\" \"10.128.0.116:80\" outbound|80|ga|myapp.default.svc.cluster.local - 10.128.0.16:443 xxx.xxx.xxx.xxx:62257\n"
Application Pod's istio-proxy log.
"[2019-01-22T09:16:17.048Z] \"GET /my/spp/images.pngHTTP/1.1\" 200 - 0 3113 0 0 \"xxx.xxx.xxx.xxx\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36\" \"c0abe3be-1153-45c5-bd8e-067ab597feb4\" \"my.app.com\" \"127.0.0.1:80\" inbound|80||myapp.default.svc.cluster.local - 10.128.0.116:80 xxx.xxx.xxx.xxx:0\n"
nginx log.
{
"uri": "/my/app/image.png",
"host": "my.app.com",
"requestTime": "0.000",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36",
"xForwardedProto": "https",
"user": "",
"protocol": "HTTP/1.1",
"bodyByteSize": "3113",
"method": "GET",
"remoteAddress": "127.0.0.1",
"upstreamResponseTime": "",
"request": "GET /my/app/images.png HTTP/1.1",
"referrer": "https://my.app.com/",
"status": "200",
"xForwardedFor": "xxx.xxx.xxx.xxx"
}
Looking at this log I think that Ingressgateway is dropping the response from Pod.
UT
in the proxy's log means that a timeout occurred:
UT: Upstream request timeout in addition to 504 response code.
Try to increase connection timeout by specifying Connection Pool Settings in a Destination Rule:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: myapp
namespace: default
spec:
host: myapp.default.svc.cluster.local
trafficPolicy:
connectionPool:
tcp:
connectTimeout: 10s