Kafka Streams Local State Store persistence with Persistent Volumes on Kubernetes/OpenShift

11/25/2020

I currently have a Kafka Streams application that has a GlobalKTable materialized as a state store and persisted locally on disk. When I deploy this application to OpenShift/Kubernetes it only has access those resources within that specific pod.

What's the best way to create PVCs and then update the deployment or deployment configs dynamically to reference those stored rocksdb logs? As it is possible for those individual pod instances to go down or to have a new build triggered through a CI/CD pipeline I don't want to edit the individual pod yamls. By default I believe the state stores are persisted locally in /tmp/kafka-streams/<application-name>.

-- animusdx
apache-kafka
apache-kafka-streams
kubernetes
kubernetes-pvc
openshift

0 Answers