Istio ingressgateway 503 upstream errors when only allow mTLS traffic
1/16/2021
I am using Istio 1.8.0 with on-prem k8s v1.19..We have several microservices running where I am using STRICT mode for peerauthentication. And I can verify that if I use PERMISSIVE mode I did not receive any 503 errors.
I really get stuck to find any solution cause I do not want to use PERMISSIVE mode as recommended.
Here is the log for istio ingressgateway.
$kubectl logs -f istio-ingressgateway-75496c97df-44g6l -n istio-system
[2021-01-16T07:28:51.852Z] "GET /config HTTP/1.1" 503 URX "-" 0 95 57 57 "95.0.145.40,10.6.0.21" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36" "foo-bar.example.com" "10.6.25.34:3000" outbound|80||oneapihub-ui-dev.hub-dev.svc.cluster.local 10.6.5.216:46364 10.6.5.216:8080 10.6.0.21:14387 - -Here is the log for my ui microservice.. I did not catch any 503 errors for rest of services
$ kubectl get ns --show-labels
NAME STATUS AGE LABELS
hub-dev Active 2d19h istio-injection=enabled
istio-system Active 3d5h istio-injection=disabled
$ kubectl get pods -n hub-dev
NAME READY STATUS RESTARTS AGE
oneapihub-api-dev-79dff67cdb-hx754 3/3 Running 0 15h
oneapihub-auth-dev-76cfcb6cb4-74ljq 3/3 Running 0 15h
oneapihub-backend-dev-d76799bcd-bmwjn 2/2 Running 0 15h
oneapihub-cronjob-dev-6879dbf9b8-wvpnp 3/3 Running 0 15h
oneapihub-mp-dev-864794d446-cfqj7 3/3 Running 0 15h
oneapihub-ui-dev-67d7bb6779-8z4xt 2/2 Running 0 14h
redis-hub-master-0 2/2 Running 0 15h
$ kubectl logs -f oneapihub-ui-dev-67d7bb6779-8z4xt -n hub-dev -c istio-proxy
[2021-01-15T14:17:24.698Z] "GET /config HTTP/1.1" 503 URX "-" 0 95 65 64 "95.0.145.40,10.6.0.19" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36" "foo-bar.example.com" "10.6.25.241:3000" outbound|80||oneapihub-ui-dev.hub-dev.svc.cluster.local 10.6.5.216:37584 10.6.5.216:8080 10.6.0.19:31138 - -
[2021-01-15T14:17:24.817Z] Here is peerauthentication
$ kubectl get peerauthentication -n istio-system -o yaml
apiVersion: v1
items:
- apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"security.istio.io/v1beta1","kind":"PeerAuthentication","metadata":{"annotations":{},"name":"default","namespace":"istio-system"},"spec":{"mtls":{"mode":"STRICT"}}}
name: default
namespace: istio-system
spec:
mtls:
mode: STRICT
Here is the service.yaml of my frontend app.
capel0068340585:~ semural$ kubectl get svc -n hub-dev
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
oneapihub-ui-dev ClusterIP 10.254.47.95 <none> 80/TCP 48m
# Source: oneapihub-ui/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: RELEASE-NAME-oneapihub-ui
labels:
app.kubernetes.io/name: oneapihub-ui
app.kubernetes.io/instance: RELEASE-NAME
app.kubernetes.io/managed-by: Helm
helm.sh/chart: oneapihub-ui-0.1.0
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 3000
protocol: TCP
name: http
Here is the Gateway and VS that I created in same namepace where microservices are running.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: hub-gateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- foo-bar.example.com
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: hub
spec:
hosts:
- foo-bar.example.com
gateways:
- hub-gateway
http:
- route:
- destination:
host: oneapihub-ui-dev.hub-dev.svc.cluster.local
port:
number: 80Here is the proxy config for ingressgateway
$ istioctl proxy-config route istio-ingressgateway-75496c97df-44g6l -n istio-system -o json
[
{
"name": "http.80",
"virtualHosts": [
{
"name": "foo-baexample.com:80",
"domains": [
"foo-bar.example.com",
"foo-bar.example.com:*"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "outbound|80||oneapihub-ui-dev.hub-dev.svc.cluster.local",
"timeout": "0s",
"retryPolicy": {
"retryOn": "connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes",
"numRetries": 2,
"retryHostPredicate": [
{
"name": "envoy.retry_host_predicates.previous_hosts"
}
],
Here are the outputs of istioctl describe pods.
$ istioctl x describe pod oneapihub-api-dev-78fbccf48c-5hb4c -n hub-dev
Pod: oneapihub-api-dev-78fbccf48c-5hb4c
Pod Ports: 50004 (oneapihub-api), 15090 (istio-proxy)
--------------------
Service: oneapihub-api-dev
Port: http 80/HTTP targets pod port 50004
$ istioctl x describe pod oneapihub-auth-dev-7f8998cd69-gzmnm -n hub-dev
Pod: oneapihub-auth-dev-7f8998cd69-gzmnm
Pod Ports: 50002 (oneapihub-auth), 15090 (istio-proxy)
--------------------
Service: oneapihub-auth-dev
Port: http 80/HTTP targets pod port 5000
$ istioctl x describe pod oneapihub-backend-dev-849b4bcd5d-fcm4l -n hub-dev
Pod: oneapihub-backend-dev-849b4bcd5d-fcm4l
Pod Ports: 50001 (oneapihub-backend), 15090 (istio-proxy)
--------------------
Service: oneapihub-backend-dev
Port: http 80/HTTP targets pod port 50001
$ istioctl x describe pod oneapihub-cronjob-dev-58b64d9c68-lv5bk -n hub-dev
Pod: oneapihub-cronjob-dev-58b64d9c68-lv5bk
Pod Ports: 50005 (oneapihub-cronjob), 15090 (istio-proxy)
--------------------
Service: oneapihub-cronjob-dev
Port: http 80/HTTP targets pod port 50005
$ istioctl x describe pod oneapihub-mp-dev-74fd6ffc9f-65gh5 -n hub-dev
Pod: oneapihub-mp-dev-74fd6ffc9f-65gh5
Pod Ports: 50003 (oneapihub-mp), 15090 (istio-proxy)
--------------------
Service: oneapihub-mp-dev
Port: http 80/HTTP targets pod port 50003
$ istioctl x describe pod oneapihub-ui-dev-7fd56f747c-nr5fk -n hub-dev
Pod: oneapihub-ui-dev-7fd56f747c-nr5fk
Pod Ports: 3000 (oneapihub-ui), 15090 (istio-proxy)
--------------------
Service: oneapihub-ui-dev
Port: http 80/HTTP targets pod port 3000
Exposed on Ingress Gateway http://53.6.48.168
VirtualService: hub
1 HTTP route(s)
$ istioctl x describe pod redis-hub-master-0 -n hub-dev
Pod: redis-hub-master-0
Pod Ports: 6379 (redis-hub), 15090 (istio-proxy)
Suggestion: add 'version' label to pod for Istio telemetry.
--------------------
Service: redis-hub-headless
Port: redis 6379/Redis targets pod port 6379
--------------------
Service: redis-hub-master
Port: redis 6379/Redis targets pod port 6379Here is my destination rule
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: backend-destination-rule
namespace: hub-dev
spec:
host: oneapihub-backend-dev.hub-dev.svc.cluster.local
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
sni: oneapihub-backend-dev.hub-dev.svc.cluster.local-- semural
istio
kubernetes
Similar Questions
2 Answers
1/18/2021
The solution worked for me to add rewrite:authority to the virtual service
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: hub
spec:
hosts:
- foo-example.com
gateways:
- hub-gateway
http:
- route:
- destination:
host: oneapihub-ui-dev.hub-dev.svc.cluster.local
port:
number: 80
rewrite:
authority: oneapihub-backend-dev.hub-dev.svc.cluster.local-- semural
Source: StackOverflow
1/16/2021
If it works with Pemissive and you get 503 only for Strict, then the reason for that is you are sending TLS traffic to plain text http port.
I believe the target POD (Upstream) doesn't have sidecar enabled and your actual service running in the POD is accepting plain text http instead of TLS.
To resolve this, you can inject side car in these pods as well, or you will have to make your actual target service mTLS with all proper certificate configurations.
In the end, if you are setting peer authentication as Strict then you can't have plain text traffic in and out of your mesh.
-- nitgeek
Source: StackOverflow