this question is similar to Kubernetes PetSet DNS not working but (I believe) distinct. My problem is: I want to use a Kubernetes PetSet to run a sharded database (RethinkDB). I need to pass each shard the dns address of another shard in the database, so that the shards can connect to each other run as a cluster. I also need other services to connect to the database and query it, and I'd like to do that through a k8s NodePort service (I think that if other pods connect to RethinkDB through a service, each client pod will connect to a random Rethink pod, providing a basic kind of load balancing. Using a NodePort service also means I can connect to the Rethink admin console from outside the cluster).

I believe Kubernetes should assign each RethinkDB shard a consistent domain name, and I should be able to pass each shard e.g. rethink-0.rethink-service.default.svc.cluster.local for clustering. However, I've tried two ways of configuring my PetSet and neither seems to assign the domain name rethink-0.rethink-service.default.svc.local:

1) I created a non-headless service for talking to the PetSet and that's it. In this configuration, the the only rethink pet I create seems to be getting a random name:

$ kc get all
NAME       DESIRED   CURRENT   READY     AGE
rc/etcd    1         1         1         46s
rc/pachd   1         1         1         46s
NAME                  CLUSTER-IP   EXTERNAL-IP   PORT(S)                        AGE
svc/etcd              10.0.0.206   <none>        2379/TCP,2380/TCP              46s
svc/kubernetes        10.0.0.1     <none>        443/TCP                        3d
svc/pachd             10.0.0.176   <nodes>       650/TCP,651/TCP                46s
svc/rethink-service   10.0.0.3     <nodes>       8080/TCP,28015/TCP,29015/TCP   46s
NAME             READY     STATUS    RESTARTS   AGE
po/etcd-x02ou    1/1       Running   0          46s
po/pachd-cqdus   1/1       Running   1          46s
po/rethink-0     1/1       Running   0          46s
  info: 2 completed object(s) was(were) not shown in pods list. Pass --show-all to see all objects.


NAME                                 STATUS    VOLUME             CAPACITY   ACCESSMODES   AGE
pvc/rethink-volume-claim-rethink-0   Bound     rethink-volume-0   1Gi        RWO           46s

$ kubectl run -i --tty --image ubuntu dns-test --restart=Never /bin/sh
...
# nslookup -type=srv rethink-service.default.svc.cluster.local
Server:     10.0.0.10
Address:    10.0.0.10#53

rethink-service.default.svc.cluster.local   service = 10 100 0 3231383531646337.rethink-service.default.svc.cluster.local.

Here my RethinkDB pet seems to get the name 3231383531646337.rethink-service.default.svc.cluster.local

2) I created both a non-headless service (for external services to talk to Rethink) and a headless service (for domain name assignment) and I still seem to get random DNS names:

$ kc get all
NAME       DESIRED   CURRENT   READY     AGE
rc/etcd    1         1         1         6m
rc/pachd   1         1         1         6m
NAME                   CLUSTER-IP   EXTERNAL-IP   PORT(S)                        AGE
svc/etcd               10.0.0.59    <none>        2379/TCP,2380/TCP              6m
svc/kubernetes         10.0.0.1     <none>        443/TCP                        3d
svc/pachd              10.0.0.222   <nodes>       650/TCP,651/TCP                6m
svc/rethink-headless   None         <none>                                       6m
svc/rethink-service    10.0.0.30    <nodes>       8080/TCP,28015/TCP,29015/TCP   6m
NAME             READY     STATUS    RESTARTS   AGE
po/etcd-anc7v    1/1       Running   0          6m
po/pachd-i1anr   1/1       Running   1          6m
po/rethink-0     1/1       Running   0          6m
  info: 2 completed object(s) was(were) not shown in pods list. Pass --show-all to see all objects.


NAME                                 STATUS    VOLUME             CAPACITY   ACCESSMODES   AGE
pvc/rethink-volume-claim-rethink-0   Bound     rethink-volume-0   1Gi        RWO           6m

$ kubectl run -i --tty --image ubuntu dns-test --restart=Never /bin/sh
...
# nslookup -type=srv rethink-service.default.svc.cluster.local
Server:     10.0.0.10
Address:    10.0.0.10#53

rethink-service.default.svc.cluster.local   service = 10 100 0 6638393531396237.rethink-service.default.svc.cluster.local.

Here my RethinkDB pet seems to get the name 6638393531396237.rethink-service.default.svc.cluster.local which still seems arbitrary.

My basic questions are: Do I need to connect the nodes to a headless service, in addition to my non-headless NodePort service, to get stable DNS addresses? Can I even have two services for the same set of nodes? Why do neither of these setups give rethink-0 the domain name rethink-0.rethink-<something>.default.svc.cluster.local?

Thank you so much for your help!!!

Edit: two updates:

1) Here's the complete k8s manifest I'm using. It's long, but I'd be happy to extract certain parts if that's helpful: http://pastebin.com/nm73Xtxi

2) I can't seem to do any DNS resolution related to my headless RethinkDB service, rethink-headless:

# nslookup rethink-headless.default
Server:     10.0.0.10
Address:    10.0.0.10#53

** server can't find rethink-headless.default: NXDOMAIN

# nslookup rethink-headless
Server:     10.0.0.10
Address:    10.0.0.10#53

** server can't find rethink-headless: SERVFAIL