I have an event-sourced CQRS architecture hosted in kubernetes. I have a single writer (the "denormalizer service") that listens to events and writes the denormalized views to a datastore. These views are then served by a separate view service. When the denormalizer image is updated through a deployment with new projections it replays all events from the beginning and writes the new views to a different datastore.
So I need 2 instances of denormalizer, one with the old code, and another replaying the events through the new code. When the new code is finished replaying I need to: 1) signal to the view service to switch to the newly written datastore and then, 2) bring down the old denormalizer deployment as it is no longer required.
Problem is, (to my limited knowledge) kubernetes seems ill-equipped to deal with this scenario.
Any idea how I would do something like this?
I don't know the specifics of your system, but two solutions come to my mind.
You can define a readinessProbe for your writer service. Make it report the service is ready, when the rewrite is done. Then, the rolling updater will know when to shut down the old version of the writer and start serving traffic to the new one. The only thing you'd need to do more is notify the viewer to switch to new data source. This could be done by the writer calling some API on the viewer service.
You can create a special process that will execute the procedure you described using Kubernetes API. It is more work than the first solution, but gives you more control over the whole process. It would observe your repository if there are new versions of writers, if yes, it would start a new service, wait for it to be ready, kill the old writer and notify the viewer.