Kubernetes internal pods connecting to Mosquitto pod

4/16/2020

I'm running a Mosquitto pod (docker.io/jllopis/mosquitto:v1.6.8-2) on an AKS instance (incidentally, using HTTP auth backend with the plugin) and have exposed that through a K8s Service. Looking at the logs for the broker I can see constant (multiple times at the same timestamp) sets of records like this:

1587048303: New connection from 10.240.0.6 on port 8883.
1587048303: New connection from 10.240.0.6 on port 1883.
1587048303: New connection from 10.240.0.6 on port 1883.
1587048305: Socket error on client <unknown>, disconnecting.
1587048305: Socket error on client <unknown>, disconnecting.

These come from different IP addresses but all within the same range; and checking using kubectl get pods --all-namespaces -o wide I can see that they are internal k8s processes, such as more-fs-watchers-sb64w, in the kube-system namespace.

What are all these doing and how can I stop them bombarding the broker? Why are they doing it? And could this be affecting other MQTT clients, legitimate ones, that are reporting intermittent connection problems?

-- MrShoes
azure-aks
kubernetes
mosquitto

1 Answer

4/16/2020

I suspect that you are running the more-fs-watcher daemonset in your cluster.

This was vaguely recommended to go around the following issue: https://github.com/Azure/AKS/issues/772

Note that the issue is now fixed and live in the latest AKS cluster, so it should be safe to remove the more-fs-watcher DaemonSet.

-- djsly
Source: StackOverflow