Istio mTLS issue with bookinfo example application

2/21/2019

I've installed Istio 1.1 RC on a fresh GKE cluster, using Helm, and enabled mTLS (some options omitted like Grafana and Kiali):

helm template istio/install/kubernetes/helm/istio \
  --set global.mtls.enabled=true \
  --set global.controlPlaneSecurityEnabled=true \
  --set istio_cni.enabled=true \
  --set istio-cni.excludeNamespaces={"istio-system"} \
  --name istio \
  --namespace istio-system >> istio.yaml

kubectl apply -f istio.yaml

Next, I installed the Bookinfo example app like this:

kubectl label namespace default istio-injection=enabled
kubectl apply -f istio/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f istio/samples/bookinfo/networking/bookinfo-gateway.yaml
kubectl apply -f istio/samples/bookinfo/networking/destination-rule-all-mtls.yaml

Then I've gone about testing by following the examples at: https://istio.io/docs/tasks/security/mutual-tls/

My results show the config is incorrect, but the verification guide above doesn't provide any hints about how to fix or diagnose issues. Here's what I see:

istio/bin/istioctl authn tls-check productpage.default.svc.cluster.local
Stderr when execute [/usr/local/bin/pilot-discovery request GET /debug/authenticationz ]: gc 1 @0.015s 6%: 0.016+1.4+1.0 ms clock, 0.064+0.31/0.45/1.6+4.0 ms cpu, 4->4->1 MB, 5 MB goal, 4 P
gc 2 @0.024s 9%: 0.007+1.4+1.0 ms clock, 0.029+0.15/1.1/1.1+4.3 ms cpu, 4->4->2 MB, 5 MB goal, 4 P

HOST:PORT                                      STATUS     SERVER     CLIENT     AUTHN POLICY     DESTINATION RULE
productpage.default.svc.cluster.local:9080     OK         mTLS       mTLS       default/         default/istio-system

This appears to show that mTLS is OK. And the previous checks all pass, like checking the cachain is present, etc. The above check passes for all the bookinfo components.

However, the following checks show an issue:

1: Confirm that plain-text requests fail as TLS is required to talk to httpbin with the following command: 

kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl http://productpage:9080/productpage -o /dev/null -s -w '%{http_code}\n'

200 <== Error.  Should fail.

2: Confirm TLS requests without client certificate also fail:
kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl https://productpage:9080/productpage -o /dev/null -s -w '%{http_code}\n' -k

000 <=== Correct behaviour
command terminated with exit code 35 

3: Confirm TLS request with client certificate succeed:
kubectl exec $(kubectl get pod -l app=productpage -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl https://productpage:9080/productpage -o /dev/null -s -w '%{http_code}\n' --key /etc/certs/key.pem --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem -k

000 <=== Incorrect. Should succeed.
command terminated with exit code 35

What else can I do to debug my installation? I've followed the installation process quite carefully. Here's my cluster info:

Kubernetes master is running at https://<omitted>
calico-typha is running at https://<omitted>/api/v1/namespaces/kube-system/services/calico-typha:calico-typha/proxy
GLBCDefaultBackend is running at https://<omitted>/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy
Heapster is running at https://<omitted>/api/v1/namespaces/kube-system/services/heapster/proxy
KubeDNS is running at https://<omitted>/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://<omitted>/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

Kubernetes version: 1.11.7-gke.4

I guess I'm after either a more comprehensive guide or some specific things I can check.

Edit: Additional info: Pod status:

default ns:
NAME                              READY   STATUS    RESTARTS   AGE
details-v1-68868454f5-l7srt       2/2     Running   0          3h
productpage-v1-5cb458d74f-lmf7x   2/2     Running   0          2h
ratings-v1-76f4c9765f-ttstt       2/2     Running   0          2h
reviews-v1-56f6855586-qszpm       2/2     Running   0          2h
reviews-v2-65c9df47f8-ztrss       2/2     Running   0          3h
reviews-v3-6cf47594fd-hq6pc       2/2     Running   0          2h

istio-system ns:
NAME                                                        READY   STATUS      RESTARTS   AGE
grafana-7b46bf6b7c-2qzcv                                    1/1     Running     0          3h
istio-citadel-5bf5488468-wkmvf                              1/1     Running     0          3h
istio-cleanup-secrets-release-1.1-latest-daily-zmw7s        0/1     Completed   0          3h
istio-egressgateway-cf8d6dc69-fdmw2                         1/1     Running     0          3h
istio-galley-5bcd455cbb-7wjkl                               1/1     Running     0          3h
istio-grafana-post-install-release-1.1-latest-daily-vc2ff   0/1     Completed   0          3h
istio-ingressgateway-68b6767bcb-65h2d                       1/1     Running     0          3h
istio-pilot-856849455f-29nvq                                2/2     Running     0          2h
istio-policy-5568587488-7lkdr                               2/2     Running     2          3h
istio-sidecar-injector-744f68bf5f-h22sp                     1/1     Running     0          3h
istio-telemetry-7ffd6f6d4-tsmxv                             2/2     Running     2          3h
istio-tracing-759fbf95b7-lc7fd                              1/1     Running     0          3h
kiali-5d68f4c676-qrxfd                                      1/1     Running     0          3h
prometheus-c4b6997b-6d5k9                                   1/1     Running     0          3h

Example destinationrule:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  creationTimestamp: "2019-02-21T15:15:09Z"
  generation: 1
  name: productpage
  namespace: default   
spec:
  host: productpage
  subsets:
  - labels:
      version: v1
    name: v1
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
-- Mick Sear
istio
kubernetes

1 Answer

2/22/2019

If you are using Istio 1.1 RC, you should be looking at the docs at https://preliminary.istio.io/ instead of https://istio.io/. The preliminary.istio.io site is always the working copy of the docs, corresponding to the next to be Istio release (1.1 currently).

That said, those docs are currently changing a lot day-to-day as they are being cleaned up and corrected during final testing before 1.1 is released, probably in the next couple of weeks.

A possible explanation for the plain text http request returning 200 in you test is that you may be running with permissive mode.

-- Frank B
Source: StackOverflow