I am running a kubernetes cluster with NFS NAStorage, and when I mount volumes they get a .snapshot directory created at the mountpoint. This causes problems for example when using Helm Charts, as these don't expect an unknown Read Only directory in certain paths (e.g. chown ... <dir> can fail, crashing the container).

When installing the Graylog Helm Chart, I noticed the initContainer for the graylog pod crashing due to chown: ... Read-only file system after running the following chown line:

chown -R 1100:1100 /usr/share/graylog/data/

where the following volume is mounted:

...
volumeMounts:
  - mountPath: /usr/share/graylog/data/journal
    name: journal
...

I tried working around this by modifying the command to fail "quietly" by making it run : upon failure:

chown -fR 1100:1100 /usr/share/graylog/data/ || :

This made the initContainer succeed, but now the main container crashes instead, this time due to the mere presence of the .snapshot dir.

...
kafka.common.KafkaException: Found directory /usr/share/graylog/data/journal/.snapshot, '.snapshot' is not in the form of topic-partition
If a directory does not contain Kafka topic data it should not exist in Kafka's log directory
...

I have tried modifying the mount point of the volume, too, moving it up one level, but this causes new issues:

...
volumeMounts:
  - mountPath: /usr/share/graylog/data
    name: data-journal
...

com.github.joschi.jadconfig.ValidationException: Parent path /usr/share/graylog/data/journal for Node ID file at /usr/share/graylog/data/journal/node-id is not a directory

I would have expected there to be some way of disabling the creation of the .snapshot directory, ideally a way to unmount/never mount it in the first place. That, or any way to have the container properly ignore the directory entirely, to make it not interfere with the processes in the container, since it seems the very presence of it can seriously disrupt. However, I have yet to find anything of the sort, and I can't seem to find anyone having had a similar issue (the introduction of Volume Snapshots in kubernetes has not made the searching easier, to say the least).

Edit 1

I tried (semi successfully, I get the Parent path ... is not a directoryerror above) to implement subPath: journal, thus circumventing the .snapshot directory (or so I believe), but this still means potentially editing every Chart that is used in my cluster. Hopefully an alternative on a higher level can be found.

volumeMounts:
  - mountPath: /usr/share/graylog/data/journal
    name: data-journal
    subPath: journal

Edit 2

I am running a bare-metal cluster, with MetalLB and Nginx as loadbalancer+ingress controller. The storage solution is provided by a third party provider, and it is from their backup solution that the .snapshot directory is made.

My imagined workaround

Since this will mainly be a problem when using Helm Charts or other deployments where volume mounts will be more or less out of our control, I will look into applying a "kustomization" that adds a single line to each volumeMount, adding

...
subPath: mount

or something like that. By doing that, I should be separating the actual mount point in the volume and the directory that actually gets mounted in the container by one level, keepin the .snapshot directory hidden in the abstract volume object. I will post my findings and potential kustomization that may come of it, if anyone else runs into a similar problem.

If someone thinks of a more streamlined solution, it is still very welcome - I'm sure it is possible to improve upon this one.