I could not be understanding Kubernetes correctly but our application relies on a proprietary closed-source vendor that in turn relies on Solr. I've read articles on rolling updates with StatefulSets but they seem to be dependent on the application being aware and accounting for new schema versions, which we have no ability to do without decompiling and jumping through a lot of hoops. Let me describe what we're trying to do:
WebService Version 1 needs to be upgraded to WebService Version 2, this upgrade is none of our code and just the vendor code our code relies on. Think of it like updating the OS.
However WebService Version 1 relies on Solr Version 1. The managed schema is different and there are breaking changes between Solr Version 1 and 2. Both the Solr version and schemas are different. If WebService Version 1 hits Solr Version 2 it won't work, or worse run break Solr Version 2. The same is true in reverse, if we update WebService Version 2 and it gets Solr Version 1 it will break that.
The only thing I can think of is to get Kubernetes to basically spin up a pod for each version and not bring down 1 until 2 is up for both WebService and Solr.
This seems not right, am I understanding this correctly?
It seems that what the canary release strategy with Solr suggests is simply having a new StatefulSet with the same labels as the one with the previous version.
Since labels can be assigned to many objects and network-level, services route requests based on these, what will happen is that requests will be redirected to both StatefulSets, emulating the canary release model.
Following this logic, you can have a v1 StatefulSet with, say, 8 replicas and another v2 StatefulSet with 2. So, ~80% of requests should hit v1 and ~20% v2 (not actually, just to illustrate).
From there, you can play with the number of replicas of each StatefulSet until you "roll out" 100% of replicas of v2, with no downtime.
Now, this can work in your scenario if you label each duo (application + Solr version) in aforementioned way.
Each duo would receive an ~N% of requests, depending on the number of replicas it has. You can slowly decrease the number of replicas of *duo* v1 and increase the next updated version.
This approach has the downside of using more resources as you will be running two versions of your full application stack. However, there is no downtime when upgrading the whole stack and you can control the percentage of "roll out".
This is not really a problem Kubernetes can solve. First work out how you would do it by hand, then you can start working out how to automate it. If zero-downtime is a requirement, the best thing I can imagine is launching the new Solr cluster separately rather than doing an in-place upgrade, then launch the new app separately pointing at the new Solr. But you will need to work out how to sync data between the two Solr clusters in real time during the upgrade. But again, Kubernetes neither helps nor hinders here, the problems are not in launching or managing the containers, it's a logistical issue in your architecture.