I am trying to take snapshot backups with Velero in Kubernetes of a 8 node test Kafka cluster with Velero such that, if the cluster failed, we could rebuild the cluster and restore the Kafka cluster from these snapshots.

We're using Velero to do that and the snapshot and restore seems to work, but on recovery, we seem to have issues with Kafka.

Has anyone gotten snapshot backups to work with Kafka at scale (roughly 3TB dataset in Kafka)? (Given the size of the dataset, doing dumps or restores from dumps is not viable.)