I deployed three back-end services to kubernetes windows pods to ensure they communicate with each other. However, I found though containers, services, dns and end-points all are available and running but still when I try to access any of the services (Internally or externally) from one container to another it does not resolve the dns and receive “could not resolve host” or "failed to connect Network unreachable"
While trying to access staffing or aggregator-service through (Resources-service), below errors are> received:
C:\WINDOWS\system32>kubectl exec -it resources-deployment-6b98d47b5b-ts674 cmd
Microsoft Windows [Version 10.0.17763.864] (c) 2018 Microsoft Corporation. All rights reserved.
C:\app\resources>curl http://172.20.101.110:8081/swagger/index.html curl: (7) Failed to connect to 172.20.101.110 port 8081: Network unreachable
C:\app\resources>curl http://aggregator-service:8081/swagger/index.html curl: (6) Could not resolve host: aggregator-service
C:\app\resources>curl http://staffing-service:8080/swagger/index.html curl: (6) Could not resolve host: staffing-service
C:\app\resources>curl http://172.20.5.156:8080/swagger/index.html curl: (7) Failed to connect to 172.20.5.156 port 8080: Network unreachable
C:\app\resources>ping 10.91.24.250
Pinging 10.91.24.250 with 32 bytes of data: PING: transmit failed. General failure.
Code is very simple, here are the complete details.
Its a mixed environment with two windows nodes and two linux nodes. However for the time being requirement is to deploy the below applications on windows nodes only. Infrastructure is deployed on EKS (Elastic Kubernetes Service) provided by AWS.
Three backend (aspnet) services are:-
Staffing.yaml file with type LoadBalancer.
Resources.Yaml having type NodePort
Aggregator.Yaml having type NodePort
Requirement is that Resources API should be able to connect to both Aggregator and Staffing services. And Staffing API too can connect to any of the services (Resources and Aggregator)
Deployment and Services YAML file are:- Staffing.yaml file
apiVersion: apps/v1
kind: Deployment
metadata:
name: staffing-deployment
#namespace: staffing-kube-system
labels:
app: staffing-app
spec:
replicas: 2
selector:
matchLabels:
app: staffing-app
template:
metadata:
labels:
app: staffing-app
spec:
nodeSelector:
beta.kubernetes.io/os: windows
#failure-domain.beta.kubernetes.io/zone: us-east-1a
containers:
- name: staffing-app
ports:
- name: http
containerPort: 8080
image: test/staffing:v1
##########################
#Staffing.service.yaml file
---
apiVersion: v1
kind: Service
metadata:
labels:
app: staffing-app
name: staffing-service
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
spec:
ports:
-
name: http
#nodePort: 30475
port: 8080
protocol: TCP
targetPort: 8080
-
name: https
#nodePort: 30476
port: 443
protocol: TCP
targetPort: 44346
selector:
app: staffing-app
type: LoadBalancerresources.service.deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: resources-deployment
#namespace: staffing-kube-system
labels:
app: resources-app
spec:
replicas: 2
selector:
matchLabels:
app: resources-app
template:
metadata:
labels:
app: resources-app
spec:
nodeSelector:
beta.kubernetes.io/os: windows
#failure-domain.beta.kubernetes.io/zone: us-east-1a
containers:
- name: resources-app
ports:
- name: http
containerPort: 8001
image: test/resources:v1
################
#resources.service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: resources-app
name: resources-service
spec:
ports:
-
name: http
#nodePort: 30475
port: 8001
protocol: TCP
targetPort: 8001
-
name: https
#nodePort: 30476
port: 443
protocol: TCP
targetPort: 44345
selector:
app: resources-app
type: NodePortaggregator.service.deplpoyment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: aggegator-deployment
#namespace: staffing-kube-system
labels:
app: aggegator-app
spec:
replicas: 2
selector:
matchLabels:
app: aggregator-app
template:
metadata:
labels:
app: aggregator-app
spec:
nodeSelector:
beta.kubernetes.io/os: windows
#failure-domain.beta.kubernetes.io/zone: us-east-1a
containers:
- name: aggregator-app
ports:
- name: http
containerPort: 8001
image: test/aggregator:v1
################
#aggregator.service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: resources-app
name: resources-service
spec:
ports:
-
name: http
#nodePort: 30475
port: 8001
protocol: TCP
targetPort: 8001
-
name: https
#nodePort: 30476
port: 443
protocol: TCP
targetPort: 44345
selector:
app: resources-app
type: NodePortOutput of Pods and services
C:\WINDOWS\system32>kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
aggegator-deployment-64497699d-792dd 1/1 Running 0 2d12h 10.91.25.210 ip-10-91-25-125.ec2.internal <none> <none>
aggegator-deployment-64497699d-tfsww 1/1 Running 0 2d12h 10.91.24.250 ip-10-91-24-148.ec2.internal <none> <none>
resources-deployment-6b98d47b5b-7x8fv 1/1 Running 0 2d13h 10.91.25.197 ip-10-91-25-125.ec2.internal <none> <none>
resources-deployment-6b98d47b5b-ts674 1/1 Running 0 2d13h 10.91.24.122 ip-10-91-24-148.ec2.internal <none> <none>
staffing-deployment-797464966d-4j4nc 1/1 Running 0 2d16h 10.91.24.224 ip-10-91-24-148.ec2.internal <none> <none>
staffing-deployment-797464966d-mrw7x 1/1 Running 0 2d17h 10.91.24.113 ip-10-91-24-148.ec2.internal <none> <none>
C:\WINDOWS\system32>kubectl get services -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
aggregator-service NodePort 172.20.101.110 <none> 8081:32246/TCP,443:30240/TCP 2d12h app=aggregator-app
kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 21d <none>
resources-service NodePort 172.20.228.37 <none> 8001:31365/TCP,443:31686/TCP 2d13h app=resources-app
staffing-service LoadBalancer 172.20.5.156 internal-af561cd570d0d11eab0c00a891642f9b-874414568.us-east-1.elb.amazonaws.com 8080:30010/TCP,443:31327/TCP 2d17h app=staffing-app
C:\WINDOWS\system32>kubectl get pods --selector="app=resources-app"
NAME READY STATUS RESTARTS AGE
resources-deployment-6b98d47b5b-7x8fv 1/1 Running 0 2d21h
resources-deployment-6b98d47b5b-ts674 1/1 Running 0 2d21h
C:\WINDOWS\system32>kubectl get pods --selector="app=aggregator-app"
NAME READY STATUS RESTARTS AGE
aggegator-deployment-64497699d-792dd 1/1 Running 0 2d20h
aggegator-deployment-64497699d-tfsww 1/1 Running 0 2d20h
C:\WINDOWS\system32>kubectl get pods --selector="app=staffing-app"
NAME READY STATUS RESTARTS AGE
staffing-deployment-797464966d-4j4nc 1/1 Running 0 3d
staffing-deployment-797464966d-mrw7x 1/1 Running 0 3d1h
C:\WINDOWS\system32>kubectl get endpoints
NAME ENDPOINTS AGE
aggregator-service 10.91.24.250:8081,10.91.25.210:8081,10.91.24.250:44347 + 1 more... 2d20h
kubernetes 10.91.24.167:443,10.91.25.120:443 22d
resources-service 10.91.24.122:8001,10.91.25.197:8001,10.91.24.122:44345 + 1 more... 2d21h
staffing-service 10.91.24.113:44346,10.91.24.224:44346,10.91.24.113:8080 + 1 more... 3d2h
> Kubectl logs
C:\WINDOWS\system32>kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-8455f84f99-d6v75 1/1 Running 0 21d
coredns-8455f84f99-jf7tz 1/1 Running 0 21d
C:\WINDOWS\system32>kubectl logs coredns-8455f84f99-jf7tz -n kube-system
.:53
2019-11-03T14:27:30.347Z [INFO] CoreDNS-1.3.1
2019-11-03T14:27:30.347Z [INFO] linux/amd64, go1.10.8, 6b56a9c9
CoreDNS-1.3.1
linux/amd64, go1.10.8, 6b56a9c9
2019-11-03T14:27:30.347Z [INFO] plugin/reload: Running configuration MD5 = 86a350d0c3832da65c10531b445c45d5
[INFO] Reloading
2019-11-25T07:00:09.649Z [INFO] plugin/reload: Running configuration MD5 = ebe4e401326d5dac294cde91064e2e76
[INFO] Reloading complete
2019-11-25T11:38:44.107Z [INFO] 10.91.24.148:57354 - 8232 "A IN . udp 17 false 512" NOERROR qr,rd,ra 92 0.000084547s
2019-11-25T11:38:44.113Z [INFO] 10.91.24.148:57364 - 33842 "A IN . udp 17 false 512" NOERROR qr,rd,ra 92 0.000059248s
2019-11-25T11:39:15.287Z [INFO] 10.91.24.148:57366 - 23694 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.003154278s
2019-11-25T11:39:15.331Z [INFO] 10.91.24.148:57366 - 23694 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.069842673s
2019-11-25T11:39:15.615Z [INFO] 10.91.25.125:61221 - 24722 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.000102425s
2019-11-25T11:39:17.575Z [INFO] 10.91.25.125:61223 - 8379 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.000096859s
2019-11-25T11:39:18.925Z [INFO] 10.91.25.125:61239 - 49217 "A IN settings-win.data.microsoft.com. udp 49 false 512" NOERROR qr,rd,ra 176 0.030653295s
2019-11-25T11:39:19.006Z [INFO] 10.91.25.125:61239 - 49217 "A IN settings-win.data.microsoft.com. udp 49 false 512" NOERROR qr,rd,ra 176 0.142995164s
2019-11-25T11:39:19.205Z [INFO] 10.91.25.125:61241 - 25186 "A IN settings-win.data.microsoft.com. udp 49 false 512" NOERROR qr,rd,ra 176 0.000088335s
2019-11-25T11:39:43.422Z [INFO] 10.91.25.125:61257 - 26043 "A IN . udp 17 false 512" SERVFAIL qr,rd,ra 17 0.013510742s
2019-11-25T11:39:44.129Z [INFO] 10.91.24.148:57368 - 382 "A IN . udp 17 false 512" SERVFAIL qr,rd,ra 17 0.013958603s
2019-11-25T11:39:44.270Z [INFO] 10.91.25.125:61261 - 53992 "A IN . udp 17 false 512" SERVFAIL qr,rd,ra 17 0.014368787s
2019-11-25T11:39:44.981Z [INFO] 10.91.24.148:57370 - 41136 "A IN . udp 17 false 512" SERVFAIL qr,rd,ra 17 0.014143535s
2019-11-25T11:40:15.649Z [INFO] 10.91.25.125:61231 - 53428 "A IN ctldl.windowsupdate.com. udp 41 false 512" NOERROR qr,rd,ra 355 0.013625947s
2019-11-25T11:40:45.838Z [INFO] 10.91.24.148:57372 - 36065 "A IN . udp 17 false 512" SERVFAIL qr,rd,ra 17 0.013761424s
2019-11-25T11:40:46.688Z [INFO] 10.91.24.148:57382 - 26755 "A IN . udp 17 false 512" SERVFAIL qr,rd,ra 17 0.01357897s
2019-11-25T11:40:51.896Z [INFO] 10.91.24.148:57388 - 607 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.033610722s
2019-11-25T11:40:52.006Z [INFO] 10.91.24.148:57388 - 607 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.11348618s
2019-11-25T11:41:46.839Z [INFO] 10.91.25.125:61227 - 9202 "A IN . udp 17 false 512" SERVFAIL qr,rd,ra 17 0.013585559s
2019-11-25T11:41:47.553Z [INFO] 10.91.24.148:57339 - 12070 "A IN . udp 17 false 512" NOERROR qr,rd,ra 92 0.001922224s
2019-11-25T11:41:48.403Z [INFO] 10.91.24.148:57341 - 52311 "A IN . udp 17 false 512" NOERROR qr,rd,ra 92 0.000091287s
2019-11-25T11:42:04.827Z [INFO] 10.91.24.148:57343 - 23836 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.028061963s
2019-11-25T11:42:04.827Z [INFO] 10.91.24.148:57343 - 23836 "A IN v10.events.data.microsoft.com. udp 47 false 512" NOERROR qr,rd,ra 284 0.004729734s
2019-11-25T11:42:13.855Z [hINFO] 10.91.24.148:57344 - 62298 "A IN ctldl.windowsupdate.com. udp 41 false 512" NOERROR qr,rd,ra 540 0.012435227sAny help on this would be appreciated. Thanks
The problem was within a definition of a service where ports defined in Kubernetes file were in conflicts with what was defined in a code.