I have the following issue with on Kubernetes pod accessing a persistent volume exclusively where apparently a file has not been removed upon shutdown of the previous pod using it:

[2019-05-25 09:11:33,980] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: Failed to acquire lock on file .lock in /opt/kafka/data/logs. A Kafka instance in another process or thread is using this directory.
    at kafka.log.LogManager$anonfun$lockLogDirs$1.apply(LogManager.scala:240)
    at kafka.log.LogManager$anonfun$lockLogDirs$1.apply(LogManager.scala:236)
    at scala.collection.TraversableLike$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.TraversableLike$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
    at kafka.log.LogManager.lockLogDirs(LogManager.scala:236)
    at kafka.log.LogManager.<init>(LogManager.scala:97)
    at kafka.log.LogManager$.apply(LogManager.scala:953)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:237)
    at io.confluent.support.metrics.SupportedServerStartable.startup(SupportedServerStartable.java:114)
    at io.confluent.support.metrics.SupportedKafka.main(SupportedKafka.java:66)

I installed Kafka using the official helm chart, so I assume that the setup of Kafka and zookeeper pods as well as either assignment to persistent volumes and claims is suited for Kubernetes.

First of all: Is this a good idea to persist the running state on the persistent volume? Since pods are supposed to be not reliable and can crash or be evicted at any point in time this method is very prone to errors. Should I consider this a bug or flaw which is worth reporting to the helm chart authors?

Since bugs exist and other software might persist it's running state on the persistent volume, I'm interested in a general best practices approach how to bring the persistent volume into a state where the pod using it can start again (in this case it sould be deleting the lock file from /opt/kafka/data/logs afaik).

So far I attempted to start a console in a container shell and try to run the command to remove the file before the pod crashes. This takes some tries and is very annoying.

I'm experiencing this with microk8s 1.14.2 (608) on Ubuntu 19.10, but I think it could happen on any Kubernetes implementation.