DNS error between service communication using Kubernetes

6/6/2021

Hello i am running 2 services on kubernetes as seen below :

import express from 'express' ### Service 1
const axios = require('axios');

const app = express()
app.use(express.json())

app.listen(4001, ()=>{
    axios.post('http://service2-clusterip-srv:4002');
    console.log('listening for questions on 4001...')
})

Service 2 :

import express from 'express'


const app = express()
app.use(express.json())

app.post('/',()=>{
  console.log('got it')
})

app.listen(4002, ()=>{
    console.log('listening for questions on 4002...')
})

For each of these i've created a deployment and a clusterip object which are running and working fine.My problem is that this post request produces the error seen below :

(node:15712) UnhandledPromiseRejectionWarning: Error: getaddrinfo EAI_AGAIN service2-clusterip-srv

Of course my cluster ip yaml file has

metadata:
  name: service2-clusterip-srv 

so it is not a typo.

There are 2 dns pods as seen below,both running:

coredns-f9fd979d6-fqnvn   1/1     Running   0          14m
coredns-f9fd979d6-r5sjz   1/1     Running   0          14m

Using kubectl logs --namespace=kube-system -l k8s-app=kube-dns i get:

[INFO] plugin/ready: Still waiting on: "kubernetes"
I0606 21:42:50.141540       1 trace.go:116] Trace[2019727887]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125 (started: 2021-06-06 21:42:28.7054856 +0000 UTC m=+0.344342401) (total time: 21.4345613s):
Trace[2019727887]: [21.4345613s] [21.4345613s] END
E0606 21:42:50.141843       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: connect: connection refused
I0606 21:42:50.159454       1 trace.go:116] Trace[939984059]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125 (started: 2021-06-06 21:42:28.706811 +0000 UTC m=+0.345667901) (total time: 21.4525652s):
Trace[939984059]: [21.4525652s] [21.4525652s] END
E0606 21:42:50.159556       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Get "https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: connect: connection refused
I0606 21:42:50.175259       1 trace.go:116] Trace[1474941318]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125 (started: 2021-06-06 21:42:28.7078366 +0000 UTC m=+0.346693501) (total time: 21.4673704s):
Trace[1474941318]: [21.4673704s] [21.4673704s] END
E0606 21:42:50.175321       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: connect: connection refused
I0606 21:42:50.138456       1 trace.go:116] Trace[2019727887]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125 (started: 2021-06-06 21:42:28.6729294 +0000 UTC m=+0.238541501) (total time: 21.4651843s):
Trace[2019727887]: [21.4651843s] [21.4651843s] END
E0606 21:42:50.138711       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Get "https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: connect: connection refused
I0606 21:42:50.156165       1 trace.go:116] Trace[939984059]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125 (started: 2021-06-06 21:42:28.6721296 +0000 UTC m=+0.237741801) (total time: 21.4839389s):
Trace[939984059]: [21.4839389s] [21.4839389s] END
E0606 21:42:50.156203       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: connect: connection refused
I0606 21:42:50.157891       1 trace.go:116] Trace[1474941318]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125 (started: 2021-06-06 21:42:28.6618082 +0000 UTC m=+0.227420301) (total time: 21.4959517s):
Trace[1474941318]: [21.4959517s] [21.4959517s] END
E0606 21:42:50.157924       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: connect: connection refused
[INFO] plugin/ready: Still waiting on: "kubernetes"

which are the logs for my two dns pods.These probably indicate that something is wrong but i cant figure it out.

Any ideas about how could i proceed?

I've been stuck in this for a while so any help would be appreciated

-- peoloustis
kubernetes

0 Answers