I try to setup pods in Kubernetes but the pods seem highly unstable. But the responses from the pod seem fully random.
This is the output of an Apache pod running in my k3s cluster (1 master, 3 workers).
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
<html><body><h1>It works!</h1></body></html>
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to host
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to host
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to host
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to host
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to host
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to host
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to host
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to host
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to host
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to host
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to host
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
<html><body><h1>It works!</h1></body></html>
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
<html><body><h1>It works!</h1></body></html>
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
<html><body><h1>It works!</h1></body></html>
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
<html><body><h1>It works!</h1></body></html>
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to host
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to host
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
<html><body><h1>It works!</h1></body></html>
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to host
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
<html><body><h1>It works!</h1></body></html>
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to host
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
<html><body><h1>It works!</h1></body></html>
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
<html><body><h1>It works!</h1></body></html>
sebastian@kobol:~$ curl -k http://192.168.30.13:30081
curl: (7) Failed to connect to 192.168.30.13 port 30081: No route to hostThe responses come from one of my worker nodes, but the other two worker nodes behave exactly the same. I would expect to have an Apache instance running on each worker node (the deployment config specifies "replicas: 3")
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: apache-deployment
namespace: webservers
labels:
app: apache
spec:
replicas: 3
selector:
matchLabels:
app: apache
template:
metadata:
labels:
app: apache
spec:
containers:
- name: apache
image: httpd:2.4
ports:
- containerPort: 80---
apiVersion: v1
kind: Service
metadata:
name: apache-svc
namespace: webservers
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
nodePort: 30081
selector:
app: apacheOn top I try to setup an Ingress using Nginx ingress controller for this Apache service. But all I get is "404 page not found" all the time.
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: webservers-ingress
namespace: webservers
annotations:
kubernetes.io/ingress.class: nginx # use the shared ingress-nginx
nginx.ingress.kubernetes.io/rewrite-target: /
#nginx.ingress.kubernetes.io/service-upstream: "true"
spec:
rules:
- host: apache.v-kube-cluster.local
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: apache-svc
port:
number: 80Can anyone give me a hint on how I can stabilize my environment?
UPDATE #1: Some information about my cluster and pods after I deployed applications. The apache- and nginx-pods and services are the interesting ones.
sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes
NAME STATUS ROLES AGE VERSION
v-k3s-worker-1 Ready <none> 13m v1.21.4+k3s1
v-k3s-worker-3 Ready <none> 8m21s v1.21.4+k3s1
v-k3s-worker-2 Ready <none> 10m v1.21.4+k3s1
v-k3s-master Ready control-plane,master 16m v1.21.4+k3s1
kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get pod --all-namespaces -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName
NAME STATUS NODE
helm-install-traefik-crd-jf848 Succeeded v-k3s-master
helm-install-traefik-9mhgg Succeeded v-k3s-master
traefik-97b44b794-trz76 Running v-k3s-master
local-path-provisioner-5ff76fc89d-h4hlt Running v-k3s-master
metrics-server-86cbb8457f-c95bt Running v-k3s-master
coredns-7448499f4d-pjg4c Running v-k3s-master
svclb-traefik-jk2vk Running v-k3s-master
svclb-traefik-bnhv9 Running v-k3s-worker-1
svclb-traefik-v8wm6 Running v-k3s-worker-2
svclb-traefik-qrg4r Running v-k3s-worker-3
ingress-nginx-admission-create-hv6r9 Succeeded v-k3s-worker-3
ingress-nginx-admission-patch-dltfr Succeeded v-k3s-worker-2
dashboard-metrics-scraper-856586f554-rsqnx Running v-k3s-worker-3
ingress-nginx-controller-8cf5559f8-5dqb8 Running v-k3s-worker-1
kubernetes-dashboard-67484c44f6-ztgpc Running v-k3s-worker-2
nginx-deployment-757fbf9d6-kzjtg Running v-k3s-worker-1
apache-deployment-5df5f65c6f-zwfqc Running v-k3s-worker-2
apache-deployment-5df5f65c6f-qbbld Running v-k3s-worker-3
nginx-deployment-757fbf9d6-k5rq7 Running v-k3s-worker-3
nginx-deployment-757fbf9d6-h5kln Running v-k3s-master
apache-deployment-5df5f65c6f-p6ckr Running v-k3s-worker-2
sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get services --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 25m
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 25m
kube-system metrics-server ClusterIP 10.43.121.217 <none> 443/TCP 25m
kube-system traefik LoadBalancer 10.43.124.27 192.168.30.10 80:30776/TCP,443:32653/TCP 24m
ingress-nginx ingress-nginx-controller-admission ClusterIP 10.43.69.180 <none> 443/TCP 5m31s
ingress-nginx ingress-nginx-controller NodePort 10.43.162.128 <none> 80:30953/TCP,443:32458/TCP 5m31s
kubernetes-dashboard kubernetes-dashboard ClusterIP 10.43.89.177 <none> 443/TCP 5m5s
kubernetes-dashboard dashboard-metrics-scraper ClusterIP 10.43.136.108 <none> 8000/TCP 5m5s
webservers apache-svc NodePort 10.43.46.218 <none> 80:30081/TCP 4m18s
webservers nginx-svc NodePort 10.43.100.206 <none> 80:30082/TCP 4m8s
sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get ingress --all-namespaces
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
webservers webservers-ingress <none> apache.v-kube-cluster.local,nginx.v-kube-cluster.local 10.0.2.15 80 4m31s
UPDATE #2: More Information about my cluster directly from my nodes. I removed all lines from the outputs that are not related to my apache pods (too much information affect readability).
Information from master node
# Information from Ingress
user@v-k3s-master:~$ curl localhost:80
404 page not found
user@v-k3s-master:~$ curl localhost:443
404 page not found
# exec
user@v-k3s-master:~$ kubectl exec --stdin --tty apache-deployment-5df5f65c6f-qqt4t -- /bin/bash
Error from server (NotFound): pods "apache-deployment-5df5f65c6f-qqt4t" not found
user@v-k3s-master:~$ kubectl exec --stdin --tty apache-deployment-5df5f65c6f-hbbp6 -- /bin/bash
Error from server (NotFound): pods "apache-deployment-5df5f65c6f-hbbp6" not found
user@v-k3s-master:~$ kubectl exec --stdin --tty apache-deployment-5df5f65c6f-x96zm -- /bin/bash
Error from server (NotFound): pods "apache-deployment-5df5f65c6f-x96zm" not found
# pods info
user@v-k3s-master:~$ kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx ingress-nginx-admission-create-pz2m9 0/1 Completed 0 19m 10.42.2.3 v-k3s-worker-2 <none> <none>
ingress-nginx ingress-nginx-admission-patch-xmhnp 0/1 Completed 0 19m 10.42.3.3 v-k3s-worker-3 <none> <none>
ingress-nginx ingress-nginx-controller-8cf5559f8-sqmqz 1/1 Running 0 19m 10.42.1.3 v-k3s-worker-1 <none> <none>
webservers apache-deployment-5df5f65c6f-qqt4t 1/1 Running 0 18m 10.42.2.7 v-k3s-worker-2 <none> <none>
webservers apache-deployment-5df5f65c6f-hbbp6 1/1 Running 0 18m 10.42.2.6 v-k3s-worker-2 <none> <none>
webservers apache-deployment-5df5f65c6f-x96zm 1/1 Running 0 18m 10.42.1.5 v-k3s-worker-1 <none> <none>
# services info
user@v-k3s-master:~$ kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get svc apache-svc
Error from server (NotFound): services "apache-svc" not found
# services info
user@v-k3s-master:~$ kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get services --all-namespaces -o wide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
ingress-nginx ingress-nginx-controller-admission ClusterIP 10.43.185.36 <none> 443/TCP 30m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx ingress-nginx-controller NodePort 10.43.124.153 <none> 80:31333/TCP,443:30349/TCP 30m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
webservers apache-svc NodePort 10.43.216.240 <none> 80:30081/TCP 29m app=apache
webservers nginx-svc NodePort 10.43.215.23 <none> 80:30082/TCP 29m app=nginx
Information from worker-1
user@v-k3s-worker-1:~$ kubectl exec --stdin --tty apache-deployment-5df5f65c6f-qqt4t -- /bin/bash
The connection to the server localhost:8080 was refused - did you specify the right host or port?
user@v-k3s-worker-1:~$ kubectl exec --stdin --tty apache-deployment-5df5f65c6f-hbbp6 -- /bin/bash
The connection to the server localhost:8080 was refused - did you specify the right host or port?
user@v-k3s-worker-1:~$ kubectl exec --stdin --tty apache-deployment-5df5f65c6f-x96zm -- /bin/bash
The connection to the server localhost:8080 was refused - did you specify the right host or port?
Information from worker-2
user@v-k3s-worker-2:~$ kubectl exec --stdin --tty apache-deployment-5df5f65c6f-qqt4t -- /bin/bash
The connection to the server localhost:8080 was refused - did you specify the right host or port?
user@v-k3s-worker-2:~$ kubectl exec --stdin --tty apache-deployment-5df5f65c6f-hbbp6 -- /bin/bash
The connection to the server localhost:8080 was refused - did you specify the right host or port?
user@v-k3s-worker-2:~$ kubectl exec --stdin --tty apache-deployment-5df5f65c6f-x96zm -- /bin/bash
The connection to the server localhost:8080 was refused - did you specify the right host or port?
Information from worker-2
user@v-k3s-worker-3:~$ kubectl exec --stdin --tty apache-deployment-5df5f65c6f-qqt4t -- /bin/bash
The connection to the server localhost:8080 was refused - did you specify the right host or port?
user@v-k3s-worker-3:~$ kubectl exec --stdin --tty apache-deployment-5df5f65c6f-hbbp6 -- /bin/bash
The connection to the server localhost:8080 was refused - did you specify the right host or port?
user@v-k3s-worker-3:~$ kubectl exec --stdin --tty apache-deployment-5df5f65c6f-x96zm -- /bin/bash
The connection to the server localhost:8080 was refused - did you specify the right host or port?