I have a simple service and pod as described below but the readiness probe fails complaining for connection refused
apiVersion: v1
kind: Service
metadata:
name: keystone-api
spec:
selector:
app: keystone
ports:
- protocol: TCP
port: 5000
targetPort: 5000
name: public
- protocol: TCP
port: 35357
targetPort: 35357
name: admin
...
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: keystone
labels:
app: keystone
spec:
replicas: 1
selector:
matchLabels:
app: keystone
template:
metadata:
labels:
app: keystone
spec:
containers:
- name: keystone
image: openio/openstack-keystone
readinessProbe:
tcpSocket:
port: 5000
env:
- name: OS_IDENTITY_ADMIN_PASSWD
value: password
- name: IPADDR
valueFrom:
fieldRef:
fieldPath: status.podIP
ports:
- containerPort: 5000
name: public
- containerPort: 35357
name: admin
Error:
Normal Pulled 37m kubelet, kind-pl Successfully pulled image "openio/openstack-keystone"
Normal Created 37m kubelet, kind-pl Created container keystone
Normal Started 37m kubelet, kind-pl Started container keystone
Warning Unhealthy 35m (x8 over 37m) kubelet, kind-pl Readiness probe failed: dial tcp 10.244.0.10:5000: connect: connection refused
This is how I launched the deployment and service kubectl apply -f application.yaml --namespace=heat
What am i missing here? Service spec
spec:
clusterIP: 10.96.162.65
ports:
- name: public
port: 5000
protocol: TCP
targetPort: 5000
- name: admin
port: 35357
protocol: TCP
targetPort: 35357
selector:
app: keystone
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
From my VM: telnet 10.96.162.65 5000 Trying 10.96.162.65...
Kubectl describe pod logs:
Namespace: heat
Priority: 0
Node: kind-control-plane/172.17.0.2
Start Time: Sun, 19 Apr 2020 16:04:36 +0530
Labels: app=keystone
pod-template-hash=8587f8dc76
Annotations: <none>
Status: Running
IP: 10.244.0.10
IPs:
IP: 10.244.0.10
Controlled By: ReplicaSet/keystone-8587f8dc76
Containers:
keystone:
Container ID: containerd://9888e62ac7df3f076bd542591a6413a0ef5b70be2c792bbf06e423b5dae89ca0
Image: openio/openstack-keystone
Image ID: docker.io/openio/openstack-keystone@sha256:62c8e36046ead4289ca4a6a49774bc589e638f46c0921f40703570ccda47a320
Ports: 5000/TCP, 35357/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Sun, 19 Apr 2020 16:08:01 +0530
Ready: True
Restart Count: 0
Readiness: tcp-socket :5000 delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
OS_IDENTITY_ADMIN_PASSWD: password
IPADDR: (v1:status.podIP)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-wf2bp (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-wf2bp:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-wf2bp
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
## Kubectl log podname logs:
10.244.0.10 - - [19/Apr/2020 11:14:33] "POST /v3/auth/tokens HTTP/1.1" 201 2161
2020-04-19 11:14:33.699 49 INFO keystone.common.wsgi [req-fc64c89f-724c-4838-bc34-3907a8f79041 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] GET http://10.244.0.10:35357/v3/services/heat
2020-04-19 11:14:33.705 49 WARNING keystone.common.wsgi [req-fc64c89f-724c-4838-bc34-3907a8f79041 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] Could not find service: heat.: ServiceNotFound: Could not find service: heat.
10.244.0.10 - - [19/Apr/2020 11:14:33] "GET /v3/services/heat HTTP/1.1" 404 90
2020-04-19 11:14:33.970 49 INFO keystone.common.wsgi [req-3589e675-8818-4b82-ad7d-c944d9e2a232 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] GET http://10.244.0.10:35357/v3/services?name=heat
10.244.0.10 - - [19/Apr/2020 11:14:34] "GET /v3/services?name=heat HTTP/1.1" 200 341
2020-04-19 11:14:34.210 49 INFO keystone.common.wsgi [req-492a3e9f-8892-4204-8ca9-c1465e28e709 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] POST http://10.244.0.10:35357/v3/endpoints
10.244.0.10 - - [19/Apr/2020 11:14:34] "POST /v3/endpoints HTTP/1.1" 201 360
10.244.0.10 - - [19/Apr/2020 11:14:38] "GET / HTTP/1.1" 300 267
2020-04-19 11:14:38.089 49 INFO keystone.common.wsgi [req-4c8952b3-7d5b-4ee3-9cf9-f736e1628448 - - - - -] POST http://10.244.0.10:35357/v3/auth/tokens
10.244.0.10 - - [19/Apr/2020 11:14:38] "POST /v3/auth/tokens HTTP/1.1" 201 2367
2020-04-19 11:14:38.737 49 INFO keystone.common.wsgi [req-ebd817f5-d473-4909-b04d-ff0e1d5badab - - - - -] POST http://10.244.0.10:35357/v3/auth/tokens
10.244.0.10 - - [19/Apr/2020 11:14:39] "POST /v3/auth/tokens HTTP/1.1" 201 2367
2020-04-19 11:14:39.635 49 INFO keystone.common.wsgi [req-b68139dc-c62f-4fd7-9cfc-e472a88b9022 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] GET http://10.244.0.10:35357/v3/services/heat
2020-04-19 11:14:39.640 49 WARNING keystone.common.wsgi [req-b68139dc-c62f-4fd7-9cfc-e472a88b9022 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] Could not find service: heat.: ServiceNotFound: Could not find service: heat.
10.244.0.10 - - [19/Apr/2020 11:14:39] "GET /v3/services/heat HTTP/1.1" 404 90
2020-04-19 11:14:39.814 49 INFO keystone.common.wsgi [req-6562f24f-f032-4150-86d9-951318918871 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] GET http://10.244.0.10:35357/v3/services?name=heat
10.244.0.10 - - [19/Apr/2020 11:14:39] "GET /v3/services?name=heat HTTP/1.1" 200 341
2020-04-19 11:14:40.043 49 INFO keystone.common.wsgi [req-6542d767-29bf-4c1a-bbd9-a81a72e106dc 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] POST http://10.244.0.10:35357/v3/endpoints
10.244.0.10 - - [19/Apr/2020 11:14:40] "POST /v3/endpoints HTTP/1.1" 201 362
Have manually created heat service
[root@keystone-8587f8dc76-rthmn /]# openstack service list
+----------------------------------+--------------+---------------+
| ID | Name | Type |
+----------------------------------+--------------+---------------+
| ec5ad9402b3b46599f3f8862e79429b3 | keystone | identity |
| 625d8b82a67d472981789f10ba37c381 | openio-swift | object-store |
| 415b33b5d45c48f6916d38f7b146953a | heat | orchestration |
+----------------------------------+--------------+---------------+
I checked with the docker instructions, and it did run, but somehow the app was not getting deployed correctly. To check with docker it is straight forward because they are using host network, so from you host you can do netstat
, and you will see that there is nothing listening on port 5000.
I accessed the container and ran the init script (keystone-v3.sh
) again, and it started to work. I did the same on kubernetes Deployment
, and it worked too.
So, this is your functional Deployment
:
apiVersion: apps/v1
kind: Deployment
metadata:
name: keystone
labels:
app: keystone
spec:
replicas: 1
selector:
matchLabels:
app: keystone
template:
metadata:
labels:
app: keystone
spec:
containers:
- name: keystone
image: openio/openstack-keystone
command: ["./keystone-v3.sh"] #<- you add this line
readinessProbe:
tcpSocket:
port: 5000
env:
- name: OS_IDENTITY_ADMIN_PASSWD
value: password
- name: IPADDR
valueFrom:
fieldRef:
fieldPath: status.podIP
ports:
- containerPort: 5000
name: public
- containerPort: 35357
name: admin
~$ kubectl get po
NAME READY STATUS RESTARTS AGE
alpine-786c6d498d-dsxfh 1/1 Running 1 11d
curler-755cc7cfff-fwz4g 1/1 Running 1 11d
keystone-6d997f4f8c-5kkxc 1/1 Running 0 26m
nginx-6db489d4b7-jlhql 1/1 Running 1 11d
~$ kubectl logs --tail 5 keystone-6d997f4f8c-5kkxc
********************************************************************************
STARTING test server keystone.server.wsgi.initialize_public_application
Available at http://keystone-6d997f4f8c-5kkxc:5000/
DANGER! For testing only, do not use in production
********************************************************************************
~$
Or you try to fix it from the image, but I guess that's not your repo right?
UPDATE
Check on this:
~$ sudo docker run -d --net=host -e IPADDR=192.168.56.102 openio/openstack-keystone
Unable to find image 'openio/openstack-keystone:latest' locally
latest: Pulling from openio/openstack-keystone
ab5ef0e58194: Pull complete
ca37595f2b63: Pull complete
878ef80688be: Pull complete
Digest: sha256:62c8e36046ead4289ca4a6a49774bc589e638f46c0921f40703570ccda47a320
Status: Downloaded newer image for openio/openstack-keystone:latest
703a05b8fdc8b7294895122b6f369a4d0a6b4582104ed360d6be68d012ea5b3c
~$ netstat -tlpn | grep 5000
NOTE: NOTHING LISTENING ON PORT 5000
~$ sudo docker ps | grep openio
703a05b8fdc8 openio/openstack-keystone "/keystone-v3.sh" 34 seconds ago Up 32 seconds quizzical_swartz
~$ sudo docker exec -it 703a05b8fdc8 bash
[root@v1-17-master /]# ls
anaconda-post.log bin dev etc home keystone-v3.log keystone-v3.sh lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
[root@v1-17-master /]# tail keystone-v3.sh
openstack endpoint create --region "$OS_OBJECTSTORE_SERVICE_REGION" 'object-store' internal "$OS_OBJECTSTORE_URL_INTERNAL"
openstack endpoint create --region "$OS_OBJECTSTORE_SERVICE_REGION" 'object-store' admin "$OS_OBJECTSTORE_URL_ADMIN"
# Demo user
openstack domain create "$OS_USER_DEMO_DOMAIN"
openstack project create "$OS_USER_DEMO_PROJECT"
openstack user create --password "$OS_USER_DEMO_PASSWD" --project "$OS_USER_DEMO_PROJECT" "$OS_USER_DEMO_USERNAME"
openstack role add --user "$OS_USER_DEMO_USERNAME" --project "$OS_USER_DEMO_PROJECT" "$OS_USER_DEMO_ROLE"
echo '> Starting Keystone public service ...'
/usr/bin/keystone-wsgi-public --port 5000
[root@v1-17-master /]# /usr/bin/keystone-wsgi-public --port 5000 &
[1] 172
[root@v1-17-master /]# exit
~$ sudo netstat -tlpn | grep 5000
tcp 0 0 0.0.0.0:5000 0.0.0.0:* LISTEN 10207/python2
TL;DR:
I've made some tests, your docker image and deployment seems really fine ,I was able to log into the pod, it was running and listening on the port.
Warning Unhealthy...: connection refused
was because it was not given enough time for the pod to start.I edited your deployment with the following lines:
readinessProbe:
tcpSocket:
port: 5000
initialDelaySeconds: 300
periodSeconds: 30
Explanation:
initialDelaySeconds
: Number of seconds after the container has started before liveness or readiness probes are initiated. Defaults to 0 seconds. Minimum value is 0.
periodSeconds
: How often (in seconds) to perform the probe. Default to 10s. Minimum value is 1s.
NOTE: During my tests I noticed that the pod takes about 5 minutes to be running, way longer than the default 10s, that's why I set it as 300 seconds.
Meaning that after 5 minutes the pod was serving on port 5000.
Add the initialDelaySeconds
line to your deployment and you should be fine.
Here is my Reproduction:
apiVersion: v1
kind: Service
metadata:
name: keystone-api
spec:
selector:
app: keystone
ports:
- protocol: TCP
port: 5000
targetPort: 5000
name: public
- protocol: TCP
port: 35357
targetPort: 35357
name: admin
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: keystone
labels:
app: keystone
spec:
replicas: 1
selector:
matchLabels:
app: keystone
template:
metadata:
labels:
app: keystone
spec:
containers:
- name: keystone
image: openio/openstack-keystone
readinessProbe:
tcpSocket:
port: 5000
initialDelaySeconds: 300
periodSeconds: 30
env:
- name: OS_IDENTITY_ADMIN_PASSWD
value: password
- name: IPADDR
valueFrom:
fieldRef:
fieldPath: status.podIP
ports:
- containerPort: 5000
name: public
- containerPort: 35357
name: admin
$ kubectl get pods -w
NAME READY STATUS RESTARTS AGE
keystone-7fd895cfb5-kqnnn 0/1 Running 0 3m28s
ubuntu 1/1 Running 0 113m
keystone-7fd895cfb5-kqnnn 1/1 Running 0 5m4s
1/1
and I describe
the pod:$ kubectl describe pod keystone-586b8948d5-c4lpq
Name: keystone-586b8948d5-c4lpq
Namespace: default
Priority: 0
Node: minikube/192.168.39.39
Start Time: Mon, 20 Apr 2020 15:02:24 +0000
Labels: app=keystone
pod-template-hash=586b8948d5
Annotations: <none>
Status: Running
IP: 172.17.0.7
IPs:
IP: 172.17.0.7
Controlled By: ReplicaSet/keystone-586b8948d5
Containers:
keystone:
Container ID: docker://8bc14d2b6868df6852967c4a68c997371006a5d83555c500d86060e48c549165
Image: openio/openstack-keystone
Image ID: docker-pullable://openio/openstack-keystone@sha256:62c8e36046ead4289ca4a6a49774bc589e638f46c0921f40703570ccda47a320
Ports: 5000/TCP, 35357/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Mon, 20 Apr 2020 15:02:26 +0000
Ready: True
Restart Count: 0
Readiness: tcp-socket :5000 delay=300s timeout=1s period=30s #success=1 #failure=3
Environment:
OS_IDENTITY_ADMIN_PASSWD: password
IPADDR: (v1:status.podIP)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-kcw8c (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-kcw8c:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-kcw8c
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/keystone-586b8948d5-c4lpq to minikube
Normal Pulling 7m12s kubelet, minikube Pulling image "openio/openstack-keystone"
Normal Pulled 7m11s kubelet, minikube Successfully pulled image "openio/openstack-keystone"
Normal Created 7m11s kubelet, minikube Created container keystone
Normal Started 7m11s kubelet, minikube Started container keystone
As you can see now there is no error.
Let me know in the comments if you have any doubt.