A Kubernetes StatefulSet (v1.8) will automagically rollover the pods it is managing to a new image if you patch the image name in the StatefulSet spec -- orchestrating this kind of update is one of its main jobs, after all. However, if deployment of a revised image fails at the image pull step (say, there was a typo in the image name, so there is no image to pull), the first pod it tries to restart with the new image gets stuck in an ImagePullBackOff state and the rollover stops. That's fine, that's pretty much what you'd want and expect. However, if you then patch the StatefulSet spec again to correct the mistake, the set is not healed -- it just sits there in the broken state -- until you (manually) delete the failed pod, at which point everything proceeds as it should and it all works its way back to a healthy state.

I'm thinking there should be a way to configure things so that this kind of recovery happens automatically when the image name is updated, just as it does for a normal deployment, rather than requiring manual intervention (especially since this is for a CI-type application, where this all needs to be automated anyway). I can't help but feeling like I'm just not finding the right option in the documentation, but poring over the docs and doing endless Google searches hasn't revealed a winning recipe. Is there a way to configure this kind of recovery to happen automatically, and if so, what is it?

Failing that, is there a graceful way to automate detection of the failed state in order to trigger a scripted deletion of the failed pod? Everything I've tried seems really kludgy, with lots of polling and ad hoc timeouts, so I'm having trouble believing that's the Right Way to do it.