In my Kubernetes based cluster I have few Java based, dockerized microservices, which are connected with each other, so every one sends and receives some information from each other. I have Kafka/Zookeeper cluster also which again connected to this microservices. I need to have some procedure like when I'm restarting active Kafka pod, another microservices should understand it and restart themselves. How to do this? Please give some example yml.
There is no Kubernetes native way to do your job. But you can make your application to do restart your container.
You can get help from containers' livenessProbe.
Kubernetes provides livenessProbe to detect where your application is live or not. liveness probes will actually attempt to restart a container if it fails.
spec:
containers:
- name: liveness
image: <image>
livenessProbe:
httpGet: # make an HTTP request
port: 8080 # port to use
path: /live # endpoint to hit
scheme: HTTP # or HTTPS
initialDelaySeconds: 3
periodSeconds: 3
failureThreshold: 1
Now, make your application to return other than 200 when you need to restart.
If you can identify from your microservices that your kafka pods are restarted, then you return other than 200 as status code from /live
http call.
If you want to call kafka service directly from liveness
probe, add following under httpGet
httpGet:
httpHeaders:
- name: Host
value: kafka-svc
Now, your application will call kafka-svc.svc:8080/live
Second way:
You can write an application which will watch your Kafka pods with kubernetes watch api, If this finds any restart in Kafka pods, then you can restart all dependent pods as you need.