Kubernetes postStart lifecycle always failing

7/20/2021

Trying to solve dependency between pods using postStart lifecycle.

Use case: micro service A should start after the start of micro service B.

For that we have added one container (curl) which will check if dependent service is up or not using curl command.

But when we add any command in postStart lifecycle hook, pods keep restarting and goes in crashlookbackoff state

Deployment.yaml :

kind: Deployment
metadata:
 name: Microservice-A-deployment
spec:
 replicas: 1
 selector:
   matchLabels:
     app: Microservice-A
 template:
   metadata:
     labels:
       app: Microservice-A
       date: 20thJune2021
     annotations:
       sidecar.istio.io/rewriteAppHTTPProbers: "false"
       proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
   spec:
     containers:
       - name: curl
         image: ewoutp/docker-nginx-curl
         imagePullPolicy: IfNotPresent
         command: [ 'sh', '-c', 'touch /tmp/healthy; echo The Pod is running && sleep 50' ]
         livenessProbe:
           exec:
             command:
               - cat
               - /tmp/healthy
           initialDelaySeconds: 15
           periodSeconds: 5
         lifecycle:
           postStart:
             exec:
               command: [ "/bin/sh", "-c", 'sleep 10;until [ $(eval curl -o -I -L -s -w "%{http_code}" http://microservice-B-api-service:9439/manage/health) -eq 200 ]; do echo "Waiting for microservice-B API";sleep 10;  done; exit 0' ]
       - name: Microservice-A
         image: microserviceA:latest
         imagePullPolicy: Always
         ports:[![enter image description here][1]][1]
           - name: port
             containerPort: 8080
         livenessProbe:
           httpGet:
             path: /actuator/health
             port: 8080
           initialDelaySeconds: 120
           periodSeconds: 30
           timeoutSeconds: 30
     imagePullSecrets:
       - name: dockersecret

Note: Reason for not using init-container: As we have implemented Istio with strict MTLS policy. https://github.com/istio/istio/issues/32039

Found below while searching for this issue on internet. enter image description here

-- Ankita Sawant
istio
istio-gateway
istio-sidecar
kubernetes
mtls

2 Answers

7/20/2021

That is because your command in postStart is sleeping for 10 seconds and your LivenessProbe is configured to fail after 5 seconds.

Maybe increase initialDelaySeconds or add a failureThreshold.

-- PSchoe
Source: StackOverflow

7/20/2021

You can use readinessProbe as well with the livenessProbe like this:

        readinessProbe:
			httpGet:
				path: /api/health
				port: 8080
		initialDelaySeconds: 10
		periodSeconds: 5
		failureThreshold: 8

		// for tcpSocket use:
			readinessProbe:
				tcpSocket:
					port: 3306 
-- KayV
Source: StackOverflow