I have an app that get data from a third-party data source, it will send data to my app automatically and I can't filter it, I can only receive all. When data arrive, my app will transmit this data to a rocketmq topic.
Now I have to make this app a container and deploy it in k8s deployment with 3 replica. But these pods will all get same data and send to the same rocketmq topic.
How do I make this app horizontal scalable without sending duplicate msg to the same rocketmq topic?
Now I have to make this app a container and deploy it in k8s deployment with 3 replica. But these pods will all get same data and send to the same rocketmq topic.
There is no request. My app connect to a server and it will send data to app by TCP. Every Pod will connect to that server.
If you want to do this with more than one instance, they need to coordinate in some way.
Leader Election pattern is a way to run multiple instances, but only one can be active (e.g. when you read from the same queue). This is a pattern to coordinate - only one instance is active at the time. So this pattern only use your replicas for higher availability.
If you want that all your replicas actively work, this can be done with techniques like sharding or partitioning. This is also how e.g. Kafka (e.g. similar to a queue) makes concurrent work from queues.
There are other ways to solve this problem as well, e.g. to implement some form of locks to coordinate - but partitioning or sharding as in Kafka is probably the most "cloud native" solution.