I've been playing around with my Istio cluster configuration and I've ended up in a state I can't debug my way out of.
I have the SDS+Gateway with a public IP configured. I have deployed the Istio HelloWorld app on port 5000. I can:
istio-proxy
on the helloworld-[rnd]
pod and curl localhost:5000/hello
- this works fineistioctl proxy-config cluster
(and such) from https://istio.io/docs/ops/troubleshooting/network-issues/ and https://istio.io/docs/ops/troubleshooting/proxy-cmd/ — all report OK to everything, SYNC:ed and suchkubectl exec istio-ingressgateway-[rnd] /bin/bash
and then curl helloworld.mynamespace:5000/hello
successfully (it returns Hello version: v2 ...
But I can't make the ingressgateway actually return anything but 503 when querying its publicly bound IP. If I query without the /hello
path, it returns 404
instead, so it's obviously trying to route to the helloworld
service/deployment and failing.
So I'm in the state where I can actually contact my helloworld
service from the Istio Ingress Gateway, when asking the gateway itself curl localhost/hello -i
, or from ourside the network curl -i http://35.x.y.z/hello
I always get 503 Service Unavailable Back
I don't have any DestinationRule nor Policy applying to helloworld
, and I have Istio in strict mTLS.
I could previously today access (other) services via the ingress gateway, but then I started cleaning things up (to the point when I only have the helloworld service VirtualService+Gateway and no others), and now it doesn't work. It should be possible to debug.
What is wrong?
Not related (that I can tell):
k exec -c istio-proxy helloworld-[rnd] -- curl http://localhost:15000/logging?level=true
, the istio-proxy
envoy doesn't receive any calls from istio-ingressgateway
at all; the traffic never leaves the ingress pod, unlike this question)First of all to use curl with SDS gateway You need to use it as described in Istio documentation.
$ curl -v -HHost:httpbin.example.com \
--resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
--cacert httpbin.new.example.com/2_intermediate/certs/ca-chain.cert.pem \
https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418
...
HTTP/2 418
...
-=[ teapot ]=-
_...._
.' _ _ `.
| ."` ^ `". _,
\_;`"---"`|//
| ;/
\_ _/
`"""`
Secondly according to Istio documentation using strict mTLS (mutual TLS) authentication policy requires both services to be running TLS communication. In Your case You are trying to access plain text (HTTP) service with Istio that is using TLS. This causes mutual TLS configuration conflict.
You can verify that with istioctl
command in this section of documentation:
The
istioctl
command provides an option for this purpose. You can do:
$ istioctl authn tls-check $CLIENT_POD httpbin.default.svc.cluster.local
HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE httpbin.default.svc.cluster.local:8000 OK mTLS mTLS default/ default/istio-system
Where
$CLIENT_POD
is the ID of one of the client service’s pods.Refer to Verify mutual TLS configuration for more information.
To resolve this issue mTLS has to be turned off for this service so that Istio accepts connection from plain text to TLS services. Follow this guide to create destination rule that allows non TLS communication for specified service
To confirm that this is causing this issue You can temporarily enable Permissive mode.
Edit:
From the link you provided in the last deployment file helloworld.yaml
there is no targetPort
and this is why nginx is unreachable.
This is how it should look:
apiVersion: v1
kind: Service
metadata:
name: helloworld
labels:
app: helloworld
spec:
ports:
- port: 5000
name: http
targetPort: 80
selector:
app: helloworld
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: helloworld-v1
labels:
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: helloworld
version: v1
template:
metadata:
labels:
app: helloworld
version: v1
spec:
terminationGracePeriodSeconds: 0
containers:
- name: helloworld
image: docker.io/istio/examples-helloworld-v1
resources:
requests:
cpu: "100m"
imagePullPolicy: IfNotPresent #Always
ports:
- containerPort: 5000
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: helloworld-v2
labels:
version: v2
spec:
replicas: 1
selector:
matchLabels:
app: helloworld
version: v2
template:
metadata:
labels:
app: helloworld
version: v2
spec:
terminationGracePeriodSeconds: 0
containers:
- name: helloworld
image: docker.io/istio/examples-helloworld-v2
resources:
requests:
cpu: "100m"
imagePullPolicy: IfNotPresent #Always
ports:
- containerPort: 5000