I am new to Kubernetes and trying to deploying kafka connector in a distributed mode on kubernetes. I am clear about the setting of the worker and connector properties, but what would the best practice for kubernetes deployment? Should I deploy two pods, each with one worker and the two workers from two pods sharing the same consumer/producer group-id? Or other configurations like two workers as two containers inside one pod? I hope my question is clear, many thanks!
Short answer:
You should deploy two or more pods with a consumer container per pod.
Longer version:
It depends on what you are trying to achieve. A pod is the smallest deployable units of computing in the kubernetes world. Two or more containers should sit alongside each other in a single pod if they are tightly coupled.
For your current use case, the benefit of using kubernetes (outside of many others) would be that you could horizontally scale the number of consumers with ease by increasing replicas, thereby increasing the number of pods. These consumers need to share the same consumer group id. (The story for producers would be similar as well)
Hope that helps!