How to run Mosquitto on Kubernetes with Istio?

10/2/2020

I am having trouble subscribing and publishing to my Mosquitto server running on my Kubernetes cluster.

I know my service and pod are working. I can subscribe and publish from inside the cluster using my mosquitto service as host.

I also have a working HTTPS gateway so I can access https://mosquittourl.com.br from outside and I can see it reaches my mosquitto container on port 1883. Container logs:

New connection from 127.0.0.1 on port 1883.
Client <unknown> disconnected due to protocol error.

I guess this is the expected behavior. Because I am accessing it via https instead of mqtt.

Now, I can't connect to the container using:

$ mosquitto_pub --url mqtt://mosquittourl.com.br:<port>/test

How do I correctly set up my Istio resources so I can subscribe and publish to my mosquitto server? Is there a way to configure Istio to accept mqtt requests and redirect them to my mosquitto service?

-- Felipe Pavan
istio
kubernetes
mosquitto

1 Answer

5/13/2021

I was recently able to get Istio setup to route TLS encrypted MQTT connections to Mosquitto running in a container on Kubernetes. You should use TLS as the protocol for the port in the Istio Gateway. Eventually Istio might support MQTT as an option like it does with gRPC and Mongo, but for now you need to use TCP for unencrypted MQTT traffic, and TLS for secure MQTT connections.

There are few things you need to make sure to configure correctly to get this working.

  1. Istio Operator Profile yaml file

    For your IstioOperator that configures the main load balancer for Istio, you want to make sure to setup the correct ports. For secure MQTT traffic, that is 8883 (technically you can use any port but 8883 is the default).

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout
  components:
    egressGateways:
      - name: istio-egressgateway
        enabled: true
        k8s:
          resources:
            requests:
              cpu: 10m
              memory: 40Mi

    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          resources:
            requests:
              cpu: 10m
              memory: 40Mi
          service:
            ports:
              - port: 15021
                targetPort: 15021
                name: status-port
              - port: 31400
                targetPort: 31400
                name: tcp
                # This is the port where sni routing happens
              - port: 15443
                targetPort: 15443
                name: tls
              - port: 8883
                targetPort: 8883
                name: mqtt-secure

    pilot:
      k8s:
        env:
          - name: PILOT_TRACE_SAMPLING
            value: "100"
        resources:
          requests:
            cpu: 10m
            memory: 100Mi

  values:
    global:
      proxy:
        resources:
          requests:
            cpu: 10m
            memory: 40Mi

    pilot:
      autoscaleEnabled: false

    gateways:
      istio-egressgateway:
        autoscaleEnabled: false
      istio-ingressgateway:
        autoscaleEnabled: false
        runAsRoot: true
  1. Istio Gateway yaml file

The important part here is to make sure you specify the name of the SSL certificate you already have in your Kubernetes secrets, and to use the same port you did in the IstioOperator, and make sure the protocol is TLS. The tls: block tells the gateway to do TLS termination using the specified SSL certificate.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: my-mqtt-ssl-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
    - port:
        number: 8883
        name: mqtt-secure
        protocol: TLS
      hosts:
        - "*"
      tls:
        credentialName: cert-my.certname.com
        mode: SIMPLE
        privateKey: sds
        serverCertificate: sds
  1. Istio Virtual Service yaml file

The final step is to setup an Istio Virtual Service that takes the traffic from the Istio Gateway and routes it to the correct pod. Update your pod name in the destination. The other aspect is to match on port 8883, and reroute the traffic to mosquitto on port 1883 since the traffic has been TLS terminated in the Istio Gateway.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: my-mqtt-vs
spec:
  hosts:
    - "*"
  gateways:
    - my-mqtt-ssl-gateway
  tcp:
    - match:
        - port: 8883
      route:
        - destination:
            host: my-mqtt-broker-pod-name
            port:
              number: 1883
-- Craxiom
Source: StackOverflow