First off a disclaimer: I have only been using Azure's Kubernetes framework for a short while so my apologies for asking what might be an easy problem.

I have two Kubernetes services running in AKS. I want these services to be able to discover each other by service name. The pods associated with these services are each given an IP from the subnet I've assigned to my cluster:

$ kubectl get pods -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP         ...
tom       1/1     Running   0          69m   10.0.2.10  ...
jerry     1/1     Running   5          67m   10.0.2.21  ...

If I make REST calls between these services using their pod IPs directly, the calls work as expected. I don't want to of course use hard coded IPs. In reading up on kube dns, my understanding is that entries for registered services are created in the dns. The tests I've done confirms this, but the IP addresses assigned to the dns entries are not the IP addresses of the pods. For example:

$ kubectl exec jerry -- ping -c 1 tom.default
PING tom.default (10.1.0.246): 56 data bytes

The IP address that is associated with the service tom is the so-called "cluster ip":

$ kubectl get services
NAME   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
tom    ClusterIP   10.1.0.246   <none>        6010/TCP   21m
jerry  ClusterIP   10.1.0.247   <none>        6040/TCP   20m

The same is true with the service jerry. The problem with these IP addresses is that REST calls using these addresses do not work. Even a simple ping times out. So my question is how can I associate the kube-dns entry that's created for a service with the pod IP instead of the cluster IP?

Based on the posted answer, I updated my yml file for "tom" as follows:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: tom
spec:
  template:
    metadata:
      labels:
        app: tom
    spec:
      containers:
      - name: tom
        image: myregistry.azurecr.io/tom:latest
        imagePullPolicy: Always
        ports:
          - containerPort: 6010
---
  apiVersion: v1
  kind: Service
  metadata:
    name: tom
  spec:
    ports:
    - port: 6010
      name: "6010"
    selector:
      app: tom

and then re-applied the update. I still get the cluster IP though when I try to resolve tom.default, not the pod IP. I'm still missing part of the puzzle.

Update: As requested, here's the describe output for tom:

$ kubectl describe service tom
Name:              tom
Namespace:         default
Labels:            <none>
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"tom","namespace":"default"},"spec":{"ports":[{"name":"6010","po...
Selector:          app=tom
Type:              ClusterIP
IP:                10.1.0.139
Port:              6010  6010/TCP
TargetPort:        6010/TCP
Endpoints:         10.0.2.10:6010

The output is similar for the service jerry. As you can see, the endpoint is what I'd expect--10.0.2.10 is the IP assigned to the pod associated with the service tom. Kube DNS though resolves the name "tom" as the cluster IP, not the pod IP:

$ kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE   IP ...
tom-b4ccbfb97-wfmjp     1/1     Running   0          15h   10.0.2.10
jerry-dd8fbf98f-8jgw7   1/1     Running   0          14h   10.0.2.20

$ kubectl exec jerry-dd8fbf98f-8jgw7 nslookup tom
Name:      tom
Address 1: 10.1.0.139 tom.default.svc.cluster.local

This doesn't really matter of course as long as REST calls are routed to the expected pod IP. I've had some success with this today:

$ kubectl exec jerry-5554b956b-9kpj7 -- wget -O - http://tom:6010/actuator/health
{"status":"UP"}

This shows that even though the name "tom" resolves to the cluster IP there is routing in place that makes sure the call gets to the pod. I've tried the same call from service tom to service jerry and that also works. Curiously, a loopback, from tom to tom, times out:

$ kubectl exec tom-5c68d66cf9-dxlmf -- wget -O - http://tom:6010/actuator/health
Connecting to tom:6010 (10.1.0.139:6010)
wget: can't connect to remote host (10.1.0.139): Operation timed out
command terminated with exit code 1

If I use the pod IP explicitly, the call works:

$ kubectl exec tom-5c68d66cf9-dxlmf -- wget -O - http://10.0.2.10:6010/actuator/health
{"status":"UP"}

So for some reason the routing doesn't work in the loopback case. I can probably get by with that since I don't think we'll need to make calls back to the same service. It is puzzling though.

Peter