I am trying to configure mtls for an emqx cluster running as statefulset in kubernetes.

I am creating the ca with cert manager.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name:  emqx-ca
spec:
  isCA: true
  commonName: emqx.mydomain.com
  secretName: emqx-ca
  usages:
    - digital signature
    - key encipherment
    - crl sign
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: emqx
spec:
  ca:
    secretName: emqx-ca

Then I mount the ca secret to the pods of the emqx statefulset.

spec:
  containers:
  - envFrom:
    - configMapRef:
        name: emqx-env
    image: emqx/emqx:4.3.8
    name: emqx
    ports: [...]
    volumeMounts:
    - mountPath: /opt/emqx/etc/certs/ca
      name: emqx-ca
  volumes:
  - name: emqx-ca
    secret:
      secretName: emqx-ca

In the configmap I have amongst the general k8s configuration the below keys.

EMQX_ALLOW_ANONYMOUS: "false"
EMQX_LISTENER__SSL__EXTERNAL__CACERTFILE: etc/certs/ca/ca.crt
EMQX_LISTENER__SSL__EXTERNAL__CERTFILE: etc/certs/ca/tls.crt
EMQX_LISTENER__SSL__EXTERNAL__KEYFILE: etc/certs/ca/tls.key
EMQX_LISTENER__SSL__EXTERNAL__VERIFY: verify_peer
EMQX_LISTENER__SSL__EXTERNAL__FAIL_IF_NO_PEER_CERT: "true"

Finally, I create a client cert.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: highbyte.emqx
  namespace: mqtt
spec:
  commonName: highbyte.emqx
  secretName: highbyte.emqx
  usages:
    - client auth
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: emqx
    kind: Issuer
    group: cert-manager.io

But when I use mqtt explorer with the generated certs and try to connect on port 8883, it's not authorized.

I can see this in the emqx logs.

2022-01-14T10:10:03.118775+00:00 [warning] mqtt-explorer-a1f484be@127.0.0.1:34298 [Channel] Client mqtt-explorer-a1f484be (Username: 'undefined') login failed for not_authorized
2022-01-14T10:10:09.893602+00:00 [warning] mqtt-explorer-a1f484be@127.0.0.1:34440 [Channel] Client mqtt-explorer-a1f484be (Username: 'foo') login failed for not_authorized

What am I missing here?