I have an application that uses raft to elect a leader out of multiple instances. These instances use the gossip protocol, so it just needs to know another instance to discover the rest.
I plan to run each instance as a kubernetes pod, with replication manage by a replication controller. I will also put a service in front of these nodes so that other apps in the cluster can talk to it.
My problem is: How can I get the pods within the replica set to discover each other without the kubernetes API? Is this possible through DNS, or does kubernetes provides some environment variables?
The solution is to use a headless service. For example, we can deploy a headless service called myservice-discovery. Because the service is headless, it does not do any load-balancing or get a cluster ip address. To get the ip addresses of the pods, you then query the DNS server for myservice-discovery.mycluster to get a list of A records.
You can also set up a second normal (non-headless) service if the pods also need to be accessible to other services and pods.