I'm looking for some advice on how to structure my kubernetes-based application. I have some data producers that read data from files and put it into a kafka topic with a couple of hundred partitions, based on a logical separation of the data. I then have some consumers that are reading data from those kafka partitions. One producer may write to multiple partitions, but one consumer will only read from one partition. Once the file is processed the producer should complete, but a consumer should live forever listening to new data on its partition. I have created individual pods for the producers and consumers, but I'm wondering how to now best organize my deployment so that I can deploy these pods en-masse. The files that producers ingest, and the partitions that consumers are assigned to are currently fed as arguments.
So what I want to be able to do is supply a list of files and kubernetes will create (and re-create if they crash) enough producers to process these files. I also want to be able to specify a set of partitions and have kubernetes create a pod for each partition. I see a lot of documentation use cases where the pods in a Service are all identical and interchangeable, but not a lot of talk about services where each pod has a different configuration, and possibly role. Thanks in advance.
The design mentioned in the question doesn't fit well in the Kubernetes architecture from my point of view. According to your question, eventually you will have hundreds of different pods that are hard to manage and that can't be scaled down.
I believe, it should work much better with AWS lambda or other serverless approach. Solution design can be problematic but result would cost you less money and would be more predictable and manageable.
As an alternative You can use Jobs or CronJobs to consume data from Kafka service on regular basis.