I have a Kubernetes cluster with a deployment which has a port range specified as an environment variable. The application that's running on top of the cluster will receive connections through the specified port range, and if none of the ports are available at the time that there is another incoming connection, then the app will just close all the current connections (crashed from the user's perspective).

I am not developing this application and have no control over it. I am simply setting up and managing the cluster along with the virtual machines the cluster is on, but I have also been tasked with figuring out a way to monitor the availability of the ports in the range so that an alert can be sent.

Things I have tried:

I have written a script (down below) to check the availability of the specified port range on a single VM. The problem is that I need to be able to check the ports for all the VMs continuously as the cluster scales up and down.

The only idea I can think of for handling such a dynamic environment is having a port monitoring application that resembles my script inside the cluster that will specifically carry out this monitoring for the VMs that the other application is running on.

The problem though is when checking the ports remotely the status is closed no matter if the port is in use or not. I have tried nc -zv <VM_ip> <port> and the equivalent of checking a remote port in nmap. The port range is usually in the 16 thousands or 30 thousands.

When checking the ports locally with ss -pu state all though, the ports that aren't in use will not show up. The ports that are in use will appear and their status being "UNCONN", which I'm assuming is unconnected.

This script, as mentioned before, is able to check if a port in the specified range is being used or not on the local machine. Note, I was planning on running this script as a background process which is why it's in an infinite while loop.

#!/bin/bash

PORT_START=16384
PORT_END=16394
PORT_RANGE=$((PORT_END - PORT_START))

while true; do
    for (( i=0; i<=PORT_RANGE; i++ )); do
        port=$((PORT_START + i))
        if lsof -Pi :${port} > /dev/null ; then
            echo "Port ${port} is being used" >> test.txt
        fi
    done
done