We have a Rails app that we deploy in a Docker environment (currently ECS but moving to Kubernetes). We are adding Kafka to our architecture, using Amazon's managed Kafka service.
What's the best way to add the Kafka consumer library to our Rails app? We are using rdkafka-ruby, but should we:
According to rdkafka intro:
This gem only provides a high-level Kafka consumer
As for service, it's recommended to use kafka service in separate instances. Since you are going to use Amazon MSK as a kafka service, it seems ok to add the rdkafka gem into your app.
P.S. That's how Xing used rdkafka-ruby for migrate from transactions to streams