We have written a Go application that runs in our Kubernetes clusters and publishes interesting cluster events out to a message queue.

One of the events we are interested in is detecting when a Deployment upgrade has completed successfully. i.e. A new image was deployed, and the correct number of pods have started up and passed readiness checks.

Currently we are subscribing to deployment changes through the watch api and trying to determine when the rolling upgrade of pods is finished, but this is proving surprisingly difficult to do. The current bug is dealing with Deployments who's replica count is managed by a Horizontal Pod Autoscaler.

What is the best way to detect this kind of "upgrade completed successfully" event?