I'm currently running into a kafka kubernetes pod in CrashLoopBackOff

kafka-prod-4                           1/2       CrashLoopBackOff   2640       15d

because of with the relevant error being here I believe

[ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Exiting because log truncation is not allowed for partition __consumer_offsets-5, current leader's latest offset 0 is less than replica's latest offset 4 (kafka.server.ReplicaFetcherThread)

I start looking for partition __consumer_offsets-5 as well as the replicas offsets to try to change the offset so this error doesn't appear but I'm having trouble finding partition __consumer_offsets-5 to change the offset.

kafka-run-class kafka.tools.GetOffsetShell --broker-list kafka-prod:9092 --topic sfoo-SfooEvent --time -1
sfoo-SfooEvent:0:472643
sfoo-SfooEvent:1:472390
sfoo-SfooEvent:2:170840

https://issues.apache.org/jira/browse/KAFKA-3410

https://github.com/Yolean/kubernetes-kafka/issues/98

Some of the logs I'm seeing here https://pastebin.com/hDQCefY7

Has anyone seen a solution for this error? Do I have to nuke the cluster and start from scratch again?