I'm using SCDF 2.2.1.RELEASE with the Skipper Kubernetes deployer 2.1.2.RELEASE and I'm facing a weird behaviour which I would like to understand whether it's actually intended or an issue or just a no-feature.
When I deploy a stream, I can initially specify the number of replicas per element, following this pattern:
deployer.<appName>.count=2This works great because as many instances as indicated by me are deployed into Kubernetes. Then if I get back to the stream's definition, I see that a new property has been added probably derived from the deployer.<app>.count one provided by me:
app.<appName>.spring.cloud.stream.instanceCount=2However, it's not clear anymore how to update such count, because if I try to just change the deployer.<appName>.count property to something different such as 1 (without changing the other derived property), then SCDF replies with an error and does nothing:
Package to upgrade has no difference than existing deployed/deleted package. Not upgrading.Then, if I also change the derived property app.<appName>.spring.cloud.stream.instanceCount=1, a non-expected behaviour occurs: SCDF generates a new version for the specific app, but still starts it up with the original 2 instances, completely ignoring the new value. And if I check the new definition of the stream, it shows not-synced values:
app.<appName>.spring.cloud.stream.instanceCount=1
deployer.<appName>.count=2Therefore I'm not sure whether this is intended behaviour or just an issue or event something not yet implemented, so SCDF just interprets the change of the app.<appName>.spring.cloud.stream.instanceCount as a new version, but it is always overridden by deployer.<appName>.count. It's very confusing.
Anybody maybe from the SCDF team can throw light on this matter to determine whether this is actually an issue?
Thanks!
The deployer.<appName>.count=2 is a shortcut in SCDF not only to instantiate the deployment with an initial scale-out but it certainly also automates the configuration of Spring Cloud Stream binding properties, so the consumer instances can effectively all can belong in the same consumer group (see docs).
This property is also a requirement for when partitioning on the producer side, so we will know how to partition the data based on the downstream consumer instances.
All that said, we do not track any changes to deployer's "count" property as part of the stream update workflow. We wanted to implement a new scale() operation (see: spring-cloud/spring-cloud-deployer-kubernetes/issues/159), so it will be used behind the scenes for when the "count" is changed. It is not implemented yet, though. Feel free to share your use-cases in the issue - we can review it.
For now, however, you can use the platform's autoscale or scaling features supported in kubectl or K8s APIs to scale the consumers outside of SCDF.