Pods are not restarting after initial minikube installation

7/7/2020

I am installing pods using helm on a minikube deployed in a Linux VM. Minikube is using the docker driver.

When I initially install the pod everything works but when minikube restarts the persistent volume is not accessible.

For example the helm chart for mariadb says it should be installed as the mysql user. minikube seems to ignore the helm chart and install it as the root user with 777 privilege. After the minikube restart restart the pod is started as mysql so the directory with the pid is not accessible and even if it was, the database tables are not visible.

Has anyone else seen this? The pods runs fine in a full K8S cluster so it seems to be a minikube issue.

I'm seeing the same issue with Mariadb, rabbitmq and key cloak

When it works

PersistentVolumeClaims Normal Scheduled <unknown> default-scheduler Successfully assigned default/mariadb-slave-0 to minikube Normal Pulled 2m46s kubelet, minikube Container image "docker.io/bitnami/mariadb:10.3.23-debian-10-r44" already present on machine Normal Created 2m46s kubelet, minikube Created container mariadb Normal Started 2m46s kubelet, minikube Started container mariadb 

After reboot

Name: mariadb-master-0 Namespace: default Priority: 0 Node: minikube/172.17.0.2 Start Time: Wed, 08 Jul 2020 14:43:21 -0400 Labels: app=mariadb chart=mariadb-7.6.1 component=master controller-revision-hash=mariadb-master-d95c68d44 release=mariadb statefulset.kubernetes.io/pod-name=mariadb-master-0 Annotations: <none> Status: Running IP: 172.18.0.7 IPs: IP: 172.18.0.7 Controlled By: StatefulSet/mariadb-master


Containers: mariadb: Container ID: docker://9fa565d16e4628655068eca1b585582ec8a136467f0120dae33bb91abf54eae4 Image: docker.io/bitnami/mariadb:10.3.23-debian-10-r44 Image ID: docker-pullable://bitnami/mariadb@sha256:36bed267af44b9a8d4788a47b769aa4c126967c46d3fb80aefe9fc19d82aaeb4 Port: 3306/TCP Host Port: 0/TCP State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1

dashboard logs

github.com/bitnami/bitnami-docker-mariadb mariadb 18:52:05.47 Submit issues and feature requests at github.com/bitnami/bitnami-docker-mariadb/issues mariadb 18:52:05.47 mariadb 18:52:05.48 INFO ==> ** Starting MariaDB setup ** mariadb 18:52:05.49 INFO ==> Validating settings in MYSQL_/MARIADB_ env vars mariadb 18:52:05.49 INFO ==> Initializing mariadb database mkdir: cannot create directory '/bitnami/mariadb/data': Permission denied 
-- user3645171
kubernetes
kubernetes-helm
minikube

1 Answer

7/10/2020

As I mentioned in comments you should add initContainer that prepare the volume with the right permissions.

As far as I checked in stable mariadb chart this can be edited in the database chart values.yaml.


There is description for this on bitnami mariadb github.

Adjust permissions of persistent volume mountpoint

As the image run as non-root by default, it is necessary to adjust the ownership of the persistent volume so that the container can write data into it.

By default, the chart is configured to use Kubernetes Security Context to automatically change the ownership of the volume. However, this feature does not work in all Kubernetes distributions. As an alternative, this chart supports using an initContainer to change the ownership of the volume before mounting it in the final destination.

You can enable this initContainer by setting volumePermissions.enabled to true.


Additionally there is related github issue about that.


I hope this helps.

-- Jakub
Source: StackOverflow