Our application runs jobs on Kubernetes. We are currently querying kubernetes to give feedback to the users about the statuses of jobs and once a job is completed also want to provide them the job (pod) logs. This happens often, eg every 30s. We have such high amount of jobs now, that this pod querying and handling the information takes too long.
It would be great to be informed about kubernetes events: a pod is pending, scheduled, pulling, running, succeeded/failed. Not sure what is the best way to get these events in our application, but the main questions here is:
Is there any event that a pod is terminated (successfully or in failure) or that a job is completed?
If this does not exist, is there a workaround to get to know whether a pod or job is terminated (so that we can go fetch the pod logs)?
Kubectl get events provides events happening in the cluster. For advanced scenarios you can check kubewatch.
Kubewatch is a Kubernetes watcher that currently publishes notification to available collaboration hubs/notification channels. Run it in your k8s cluster, and you will get event notifications through webhooks.
Another option would be to add a handler in preStop which sends an event before the pod/job is terminated