I have a Kafka cluster in kubernetes with a lot of test data. I want to have some/all of that test data imported into my local Kafka cluster. This, way it would be easier for me to perform tests in the local environment with actual data from kubernetes.
So, is there a way to dump for eg: 5000 messages from a kafka topic into a file and restore them into a local kafka topic ?
The way we do it (not on Kubernetes but it does not matter in this case) is:
we found these approaches simpler and more flexible that using more complex tools/frameworks like MirrorMaker.
Replicator is a commercial tool that enables you to replicate topics from one cluster to another. Similar to MirrorMaker though, it's designed to replicate entire topics, not just part of them.
You can use kafkacat with stdin
/stdout
if you just want some kind of hacky option, but things like partitioning, topic config and all that stuff that you'd want to match for accurate testing you would have to ensure gets done properly.