I am having trouble running Consul within Istio service mesh. I am using HTTPS and gossip encryption which was working as expected before injecting Envoy sidecars (with mTLS enforced). I am trying to run a 3 member Consul cluster and I am seeing the following errors on each member:

  2020/02/15 21:15:52 [WARN] consul: error getting server health from "consul-2": rpc error getting client: failed to get conn: remote error: tls: protocol version not supported
  2020/02/15 21:15:53 [WARN] consul: error getting server health from "consul-2": context deadline exceeded
  2020/02/15 21:15:53 [INFO] agent: (LAN) joining: [consul-0.cluster consul-1.cluster consul-2.cluster]
  2020/02/15 21:15:54 [WARN] consul: error getting server health from "consul-2": rpc error getting client: failed to get conn: remote error: tls: protocol version not supported
  2020/02/15 21:15:55 [WARN] consul: error getting server health from "consul-2": context deadline exceeded
  2020/02/15 21:15:56 [WARN] consul: error getting server health from "consul-2": rpc error getting client: failed to get conn: remote error: tls: protocol version not supported
  2020/02/15 21:15:56 [WARN] agent: (LAN) couldn't join: 0 Err: 3 errors occurred:
* Failed to join 10.0.2.213: EOF
* Failed to join 10.0.3.123: EOF
* Failed to join 10.0.1.11: Encryption is configured but remote state is not encrypted

  2020/02/15 21:15:56 [WARN] agent: Join LAN failed: <nil>, retrying in 30s
  2020/02/15 21:15:57 [WARN] consul: error getting server health from "consul-2": context deadline exceeded
  2020/02/15 21:15:58 [WARN] consul: error getting server health from "consul-2": rpc error getting client: failed to get conn: remote error: tls: protocol version not supported

The ports for Consul's Service are as follows:

ports:
- name: tcp-server
  port: 8300
  targetPort: tcp-server
- name: tcp-serflan
  protocol: TCP
  port: 8301
  targetPort: tcp-serflan
- name: udp-serflan
  protocol: UDP
  port: 8301
  targetPort: udp-serflan
- name: https
  port: 8501
  targetPort: https

My best guess at what is happening is that Envoy is not downgrading from L7 to L4 for the server RPC connections as mentioned in the documentation and so Consul is trying to perform the TLS handshake with the remote Envoy sidecar rather than the remote Consul.

I could either use plaintext Consul traffic with Istio mTLS or remove Envoy sidecars and keep Consul's encryption, but if possible I would like to have both encryption by Consul but within mTLS Istio mesh (in part just to see if it's possible, maybe that usecase is not something Istio supports).