I've been playing around with my Istio cluster configuration and I've ended up in a state I can't debug my way out of.
I have the SDS+Gateway with a public IP configured. I have deployed the Istio HelloWorld app on port 5000. I can:
istio-proxy on the helloworld-[rnd] pod and curl localhost:5000/hello - this works fineistioctl proxy-config cluster (and such) from https://istio.io/docs/ops/troubleshooting/network-issues/ and https://istio.io/docs/ops/troubleshooting/proxy-cmd/ — all report OK to everything, SYNC:ed and suchkubectl exec istio-ingressgateway-[rnd] /bin/bash and then curl helloworld.mynamespace:5000/hello successfully (it returns Hello version: v2 ...But I can't make the ingressgateway actually return anything but 503 when querying its publicly bound IP. If I query without the /hello path, it returns 404 instead, so it's obviously trying to route to the helloworld service/deployment and failing.
So I'm in the state where I can actually contact my helloworld service from the Istio Ingress Gateway, when asking the gateway itself curl localhost/hello -i, or from ourside the network curl -i http://35.x.y.z/hello I always get 503 Service Unavailable Back
I don't have any DestinationRule nor Policy applying to helloworld, and I have Istio in strict mTLS.
I could previously today access (other) services via the ingress gateway, but then I started cleaning things up (to the point when I only have the helloworld service VirtualService+Gateway and no others), and now it doesn't work. It should be possible to debug.
What is wrong?
Not related (that I can tell):
k exec -c istio-proxy helloworld-[rnd] -- curl http://localhost:15000/logging?level=true, the istio-proxy envoy doesn't receive any calls from istio-ingressgateway at all; the traffic never leaves the ingress pod, unlike this question)First of all to use curl with SDS gateway You need to use it as described in Istio documentation.
$ curl -v -HHost:httpbin.example.com \
--resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
--cacert httpbin.new.example.com/2_intermediate/certs/ca-chain.cert.pem \
https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418...
HTTP/2 418
...
-=[ teapot ]=-
_...._
.' _ _ `.
| ."` ^ `". _,
\_;`"---"`|//
| ;/
\_ _/
`"""`Secondly according to Istio documentation using strict mTLS (mutual TLS) authentication policy requires both services to be running TLS communication. In Your case You are trying to access plain text (HTTP) service with Istio that is using TLS. This causes mutual TLS configuration conflict.
You can verify that with istioctl command in this section of documentation:
The
istioctlcommand provides an option for this purpose. You can do:$ istioctl authn tls-check $CLIENT_POD httpbin.default.svc.cluster.localHOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE httpbin.default.svc.cluster.local:8000 OK mTLS mTLS default/ default/istio-systemWhere
$CLIENT_PODis the ID of one of the client service’s pods.Refer to Verify mutual TLS configuration for more information.
To resolve this issue mTLS has to be turned off for this service so that Istio accepts connection from plain text to TLS services. Follow this guide to create destination rule that allows non TLS communication for specified service
To confirm that this is causing this issue You can temporarily enable Permissive mode.
Edit:
From the link you provided in the last deployment file helloworld.yaml there is no targetPort and this is why nginx is unreachable.
This is how it should look:
apiVersion: v1
kind: Service
metadata:
name: helloworld
labels:
app: helloworld
spec:
ports:
- port: 5000
name: http
targetPort: 80
selector:
app: helloworld
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: helloworld-v1
labels:
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: helloworld
version: v1
template:
metadata:
labels:
app: helloworld
version: v1
spec:
terminationGracePeriodSeconds: 0
containers:
- name: helloworld
image: docker.io/istio/examples-helloworld-v1
resources:
requests:
cpu: "100m"
imagePullPolicy: IfNotPresent #Always
ports:
- containerPort: 5000
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: helloworld-v2
labels:
version: v2
spec:
replicas: 1
selector:
matchLabels:
app: helloworld
version: v2
template:
metadata:
labels:
app: helloworld
version: v2
spec:
terminationGracePeriodSeconds: 0
containers:
- name: helloworld
image: docker.io/istio/examples-helloworld-v2
resources:
requests:
cpu: "100m"
imagePullPolicy: IfNotPresent #Always
ports:
- containerPort: 5000