I have two identical pod running on two working nodes, which are served externally via a svc. as follows:
root@master1:~# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
kubia-nwjcc 1/1 Running 0 33m 10.244.1.27 worker1
kubia-zcpbb 1/1 Running 0 33m 10.244.2.11 worker2
root@master1:~# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 26h
kubia ClusterIP 10.98.41.49 <none> 80/TCP 34mBut when I try to access the svc in one of the pod, I can only get the response of the pod on the same node. When svc accesses the pod on the other node. When svc tries to access the pod on other nodes, it will return command terminated with exit code 7. correct output and bad output seem to be randomly generated, as follows
correct output
root@master1:~# k exec kubia-nwjcc -- curl http://10.98.41.49
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 23 0 23 0 0 8543 0 --:--:-- --:--:-- --:--:-- 11500
You've hit kubia-nwjccbad output
root@master1:~# kubectl exec kubia-nwjcc -- curl http://10.98.41.49
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (7) Failed to connect to 10.98.41.49 port 80: No route to host
command terminated with exit code 7
The following is the software version I am using:
v18.04v1.15.0v18.09.5The following is svc describe:
root@master1:~# kubectl describe svc kubia
Name: kubia
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=kubia
Type: ClusterIP
IP: 10.98.41.49
Port: <unset> 80/TCP
TargetPort: 8080/TCP
Endpoints: 10.244.1.27:8080,10.244.2.11:8080
Session Affinity: None
Events: <none>The following is return results with using -v=9:
root@master1:~# kubectl exec kubia-nwjcc -v=9 -- curl -s http://10.98.41.49
I0702 11:45:52.481239 23171 loader.go:359] Config loaded from file: /root/.kube/config
I0702 11:45:52.501154 23171 round_trippers.go:419] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5" 'https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc'
I0702 11:45:52.525926 23171 round_trippers.go:438] GET https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc 200 OK in 24 milliseconds
I0702 11:45:52.525980 23171 round_trippers.go:444] Response Headers:
I0702 11:45:52.525992 23171 round_trippers.go:447] Content-Type: application/json
I0702 11:45:52.526003 23171 round_trippers.go:447] Content-Length: 2374
I0702 11:45:52.526012 23171 round_trippers.go:447] Date: Tue, 02 Jul 2019 11:45:52 GMT
I0702 11:45:52.526063 23171 request.go:947] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"kubia-nwjcc","generateName":"kubia-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/kubia-nwjcc","uid":"2fd67789-c48d-4459-8b03-ac562b4a3f5c","resourceVersion":"188689","creationTimestamp":"2019-07-02T10:51:34Z","labels":{"app":"kubia"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"kubia","uid":"f3a4c457-dee4-4aec-ad73-1f0ca41628aa","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-6pgh8","secret":{"secretName":"default-token-6pgh8","defaultMode":420}}],"containers":[{"name":"kubia","image":"luksa/kubia","ports":[{"containerPort":8080,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"default-token-6pgh8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"worker1","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-03T01:35:15Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-03T01:35:20Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-03T01:35:20Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-02T10:51:34Z"}],"hostIP":"192.168.56.21","podIP":"10.244.1.27","startTime":"2019-07-03T01:35:15Z","containerStatuses":[{"name":"kubia","state":{"running":{"startedAt":"2019-07-03T01:35:19Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"luksa/kubia:latest","imageID":"docker-pullable://luksa/kubia@sha256:3f28e304dc0f63dc30f273a4202096f0fa0d08510bd2ee7e1032ce600616de24","containerID":"docker://27da556930baf857e5af92b13934dcb1b2b2f001ecab5e7b952b2bda5aa27f0b"}],"qosClass":"BestEffort"}}
I0702 11:45:52.543108 23171 round_trippers.go:419] curl -k -v -XPOST -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5" 'https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc/exec?command=curl&command=-s&command=http%3A%2F%2F10.98.41.49&container=kubia&stderr=true&stdout=true'
I0702 11:45:52.591166 23171 round_trippers.go:438] POST https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc/exec?command=curl&command=-s&command=http%3A%2F%2F10.98.41.49&container=kubia&stderr=true&stdout=true 101 Switching Protocols in 47 milliseconds
I0702 11:45:52.591208 23171 round_trippers.go:444] Response Headers:
I0702 11:45:52.591217 23171 round_trippers.go:447] Connection: Upgrade
I0702 11:45:52.591221 23171 round_trippers.go:447] Upgrade: SPDY/3.1
I0702 11:45:52.591225 23171 round_trippers.go:447] X-Stream-Protocol-Version: v4.channel.k8s.io
I0702 11:45:52.591229 23171 round_trippers.go:447] Date: Wed, 03 Jul 2019 02:29:33 GMT
F0702 11:45:53.783725 23171 helpers.go:114] command terminated with exit code 7The status of kube-system pod and the two pods providing the service is all Running, as following:
root@master1:~/k8s-yaml# kubectl get --all-namespaces pod -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default kubia-6pjz9 1/1 Running 0 5m35s 10.244.2.12 worker2 <none> <none>
default kubia-nwjcc 1/1 Running 0 16h 10.244.1.27 worker1 <none> <none>
kube-system coredns-bccdc95cf-792px 1/1 Running 4 5d19h 10.244.0.11 master1 <none> <none>
kube-system coredns-bccdc95cf-bc76j 1/1 Running 4 5d19h 10.244.0.10 master1 <none> <none>
kube-system etcd-master1 1/1 Running 8 5d19h 192.168.56.11 master1 <none> <none>
kube-system kube-apiserver-master1 1/1 Running 7 5d19h 192.168.56.11 master1 <none> <none>
kube-system kube-controller-manager-master1 1/1 Running 7 5d18h 192.168.56.11 master1 <none> <none>
kube-system kube-flannel-ds-amd64-9trbq 1/1 Running 3 5d18h 192.168.56.21 worker1 <none> <none>
kube-system kube-flannel-ds-amd64-btt74 1/1 Running 5 5d18h 192.168.56.11 master1 <none> <none>
kube-system kube-flannel-ds-amd64-swzzd 1/1 Running 0 3d20h 192.168.56.22 worker2 <none> <none>
kube-system kube-proxy-27zfk 1/1 Running 7 5d19h 192.168.56.11 master1 <none> <none>
kube-system kube-proxy-79kt9 1/1 Running 0 3d20h 192.168.56.22 worker2 <none> <none>
kube-system kube-proxy-lx4gk 1/1 Running 3 5d18h 192.168.56.21 worker1 <none> <none>
kube-system kube-scheduler-master1 1/1 Running 7 5d19h 192.168.56.11 master1 <none> <none>My expected return is: when using curl access svc, svc can randomly access the pod on different nodes as follows:
root@master1:~# kubectl get po
NAME READY STATUS RESTARTS AGE
kubia-nwjcc 1/1 Running 0 50m
kubia-zcpbb 1/1 Running 0 50m
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-zcpbb
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-zcpbb
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjccAnd now the return results of five tests are as follows:
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
command terminated with exit code 7
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
command terminated with exit code 7
This problem is solved. The official documentation of the flannel mentions that need to use --iface to specify the network card to be used when running in a vagrant type virtual machine. You can use the command kubectl edit daemonset kube-flannel-ds-amd64 -n kube-system to edit the fllannel configuration file. Then use kubectl delete pod -n kube-system <pod-name> to all flannel pods. K8s will rebuild them.
You can find detailed answers in Kubernetes with Flannel and flannel - troubleshooting.