I have an IOT application which is architected in the following way :-

There are plants which has its own sets of devices.

Now the entire pipeline is deployed in kubernetes consisting of the following units :-

1. A job which wakes up every x seconds, reads data from all plants, pushes the data to a mqtt broker.
2. An MQTT broker.
3. A subscriber service which receives data from all plants and pushes it to a timeseries database.
4. Jobs running at intervals of 5min, 15min, 1hr, 4hr, 1day and performs downsampling on the data of all the projects and pushes it to separate downsampled tables.
5. Jobs running every day to check if there was in holes/gaps in the data and tries to fill it up if possible.

Now this works fine for few Plants, but when the number of plants increases it becomes difficult perform data retrieval, push, downsampling using single service/job as it becomes too much memory intensive and chokes at multiple places. As a temporary fix scaling it vertically fixes the issue to some extent but in that case I need to put all the pods in a single machine which I am scaling vertically and scaling multiple nodes vertically is quite expensive. Hence I am planning to break down the system in a way so that I can scale horizontally and I am looking for suggestions for the possible ways I can achieve this.