I want to get your opinion on a design decision. I need to prepare a project that makes the crud operations with weather sensors and I need the communicate each weather sensor with NTCIP in the database and get information like temperature, humidity, etc.
the problem is scaling. In my architecture, I write a .net core microservice and that has a own separate database. This weather station microservice reads all weather stations from the database then communicates each sensor via IP. I open a thread for each record and these threads do the data communication over IP. My goal is to distribute database data to replicated microservice by count. For example, if there are 100 weather station records in the database and I have 2 replicas I need to distribute 50 records to the first one, and 50 to the second one. Also, I need to if I change the replica count, the distribution will be redone. I search the internet, and I found a apache helix, but it's more complicated for this operation. Please give me advice. thanks a lot
In .Net this sounds like a very good fit for Akka.Net. A reasonable "first cut" at a model would be to define an actor for each sensor and have these actors be sharded via cluster sharding, which will balance the number of sensor actors across the cluster and react to changes in cluster membership.
You might also find it useful to have the sensor actors be persistent via event sourcing, depending on where you want to go with this project (e.g. for publishing and archiving historical weather data).