I'm writing an application to be deployed to Kubernetes where I want each pod to hold a TCP connection to all the other pods in the replica set so any container can notify the other containers in some use case. If a pod is added, a new connection should be created between it and all the other pods in the replica set.
From a given pod, how do I open a TCP connection to all the other pods? I.e., how do I discover all their IP addresses?
I have a ClusterIP Service DNS name that points to these pods by selector, and that's the mechanism I've thought of trying to use so far. E.g., open a TCP connection to that DNS name repeatedly in a loop, requesting a container ID over the connection each time, pausing when I've gotten say at least three connections for each container ID. Then repeating that every minute or so to get new pods that have been added. A keepalive could be used to remove connections to pods that have gone away.
Is there a better way to do it?
If it matters, I'm writing this application in Go.
E.g., open a TCP connection to that DNS name repeatedly in a loop, requesting a container ID over the connection each time, pausing when I've gotten say at least three connections for each container ID. Then repeating that every minute or so to get new pods that have been added.
This does not sounds like a solid solution.
Kubernetes does not provide the functionality that you are looking for out-of-the-box.
Detect pods
A possible solution is to query the Kubernetes API Server, for the pods matching your selector (labels). Use client-go and look at e.g. example on how to list Pods. You may want to watch for new pods, or query regularly.
Connect to Pods
When you know the name of a pod that you want to connect to, you can use DNS to connect to pods.
If the mentioned communication between the pods is not the main functionality of this application, but rather the application requires this functionality, I would strongly suggest not re-inventing the wheel and use a 3rd party component.
Redis looks like a very good candidate, in particular Pub/Sub mechanism of Redis.
So, all of the pods will subscribe to a certain channel in Redis, and when a pod wants to notify others, it will simply publish a message on the same channel.
Redis has client libraries in many languages including Go. Also very easy to deploy to Kubernetes via its Helm chart.
A simpler solution would be to use Zookeeper to do the book keeping of currently active pods. Zookeeper is centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
It is used by components like Kafka for leader election, group membership etc.
You could use one of the recipe Group Membership or Path cache.
As soon as a pod comes up it would make a long running connection to Zookeeper. This would be used to keep track of currently available pods. When the pod goes down, the connection goes away and thus the list has only active members.