So we're deploying istio 1.0.2 with global mtls and so far it's gone well. For health checks we've added separate ports to the services and configured them as per the docs:

https://istio.io/docs/tasks/traffic-management/app-health-check/#mutual-tls-is-enabled

Our application ports are now on 8080 and health checks ports are on 8081. After doing this Kubernetes is able to do health checks and the services appear to be running normally.

However our monitoring solution cannot hit the health check port.

The monitoring application also sits in kubernetes and is currently outside the mesh. The above doc says the following:

Because the Istio proxy only intercepts ports that are explicitly declared in the containerPort field, traffic to 8002 port bypasses the Istio proxy regardless of whether Istio mutual TLS is enabled.e

This is how we have it configured. So in our case 8081 should be outside the mesh:

livenessProbe:
  failureThreshold: 3
  httpGet:
    path: /manage/health
    port: 8081
    scheme: HTTP
  initialDelaySeconds: 180
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 1
name: <our-service>
ports:
- containerPort: 8080
  name: http
  protocol: TCP
readinessProbe:
  failureThreshold: 3
  httpGet:
    path: /manage/health
    port: 8081
    scheme: HTTP
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 1

However we can't access 8081 from another pod which is outside the mesh.

For example:

curl http://<our-service>:8081/manage/health
curl: (7) Failed connect to <our-service>:8081; Connection timed out

If we try from another pod inside the mesh istio throws back a 404, which is perhaps expected.

I tried to play around with destination rules like this:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: <our-service>-health
spec:
  host: <our-service>.namepspace.svc.cluster.local
  trafficPolicy:
    portLevelSettings:
    - port:
        number: 8081
      tls:
        mode: DISABLE

But that just kills all connectivity to the service, both internally and through the ingress gateway.