K
Q

GKE (1.7.12-gke.1) Dynamic provisioning: Directory contents are created in containerFS, instead of volume FS

4/6/2018

I have a 1.7.12-gke.1 cluster.

Problem

  • When I dynamically provision volumes, I create a PVC.
  • After bootstrapping data to that PVC, I kill the container that made the data.
  • Then, i launch a second container, which binds the same PVC.

I expect that the postgres /data files from the first directory will be visible in the new container... but theyre not.

can generally see files between them when mounted, but all of the files created by postgres for some reason, under the data/ directory, arent visible in new pods.

Details

It appears somehow that this particular directorys subcontents arent stored into the volume (or at least, not in such a way that they can be mounted later in a new container from the same PVC).

I bootstrap the data directory in a volume like so:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jay-gke-pvc
spec:
  accessModes:
    - ReadWriteOnce
    - ReadOnlyMany
  resources:
    requests:
      storage: 700Mi
---
apiVersion: v1
kind: Pod
metadata:
  name: postgres-init
  labels:
    app: postgres-init
    tier: postgres-init
spec:
  serviceAccountName: postgresapp
  volumes:
  - name: postgres-persistent-vol
    persistentVolumeClaim:
      claimName: jay-gke-pvc
  initContainers:
  - name: postgres-ini-bootstrap
    image: alpine
    command: ["rm","-rf","/var/lib/postgresql/*"]
    volumeMounts:
    - name: postgres-persistent-vol
      mountPath: /var/lib/postgresql/
  containers:
  - name: postgres-ini
    image: images/my-postgres:9.6.c
    volumeMounts:
    - name: postgres-persistent-vol
      mountPath: /var/lib/postgresql/
    securityContext:
      runAsUser: 70
  restartPolicy: Never

I can see that there is plenty of data in the volume, after this pod starts:

/ $ ls -altrh /var/lib/postgresql/data/
total 140
-rw-r--r--    1 postgres postgres    1.4K Apr  6 19:24 root.crt
-rw-------    1 postgres postgres   21.6K Apr  6 19:24 postgresql.conf
drwx------    3 postgres postgres    4.0K Apr  6 19:24 pg_xlog
drwx------    2 postgres postgres    4.0K Apr  6 19:24 pg_twophase
drwx------    2 postgres postgres    4.0K Apr  6 19:24 pg_tblspc
drwx------    2 postgres postgres    4.0K Apr  6 19:24 pg_subtrans
drwx------    2 postgres postgres    4.0K Apr  6 19:24 pg_snapshots
drwx------    2 postgres postgres    4.0K Apr  6 19:24 pg_serial
drwx------    2 postgres postgres    4.0K Apr  6 19:24 pg_replslot
drwx------    4 postgres postgres    4.0K Apr  6 19:24 pg_multixact
drwx------    4 postgres postgres    4.0K Apr  6 19:24 pg_logical
-rw-------    1 postgres postgres    1.6K Apr  6 19:24 pg_ident.conf
drwx------    2 postgres postgres    4.0K Apr  6 19:24 pg_dynshmem
drwx------    2 postgres postgres    4.0K Apr  6 19:24 pg_commit_ts
drwx------    2 postgres postgres    4.0K Apr  6 19:24 pg_clog
-r--------    1 postgres postgres    1.6K Apr  6 19:24 hub-database.key
-rw-r--r--    1 postgres postgres    1.5K Apr  6 19:24 hub-database.crt
-rw-------    1 postgres postgres       4 Apr  6 19:24 PG_VERSION
drwxr-xr-x    4 root     root        4.0K Apr  6 19:24 ..
-rw-------    1 postgres postgres     874 Apr  6 19:24 postgresql.auto.conf
drwx------    8 postgres postgres    4.0K Apr  6 19:24 base
-rw-------    1 postgres postgres     940 Apr  6 19:24 pg_hba.conf
-rw-------    1 postgres postgres      85 Apr  6 19:24 postmaster.pid
-rw-------    1 postgres postgres      24 Apr  6 19:24 postmaster.opts
drwx------    2 postgres postgres    4.0K Apr  6 19:24 pg_stat
drwx------    2 postgres postgres    4.0K Apr  6 19:24 pg_notify
drwx------    2 postgres postgres    4.0K Apr  6 19:24 pg_log
drwx------   20 postgres postgres    4.0K Apr  6 19:24 .
drwx------    2 postgres postgres    4.0K Apr  6 19:24 pg_stat_tmp
drwx------    2 postgres postgres    4.0K Apr  6 19:24 global
/ $ df -h | grep postgres
/dev/sdb                975.9M      2.5M    906.2M   0% /var/lib/postgresql
/dev/sda1                94.3G     31.0G     63.2G  33% /var/lib/postgresql/data

HOWEVER When I mount this same disk in a new postgres container for a data migration, I see the following... there are no files (but the data directory, somehow, is visible).

/ $ ls /hub-db/
data/        lost+found/
/ $ ls /hub-db/data/
/ $ ls -altrh /hub-db/
total 28
drwx------    2 root     root       16.0K Apr  6 19:23 lost+found
drwxr-xr-x    2 root     root        4.0K Apr  6 19:24 data
drwxr-xr-x    4 root     root        4.0K Apr  6 19:24 .
drwxr-xr-x    1 root     root        4.0K Apr  6 19:26 ..
/ $ df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  94.3G     31.0G     63.3G  33% /
tmpfs                    14.7G         0     14.7G   0% /dev
tmpfs                    14.7G         0     14.7G   0% /sys/fs/cgroup
/dev/sdb                975.9M      2.5M    906.2M   0% /hub-db
/dev/sda1                94.3G     31.0G     63.3G  33% /dev/termination-log
/dev/sda1                94.3G     31.0G     63.3G  33% /etc/resolv.conf
/dev/sda1                94.3G     31.0G     63.3G  33% /etc/hostname
/dev/sda1                94.3G     31.0G     63.3G  33% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                    14.7G     12.0K     14.7G   0% /run/secrets/kubernetes.io/serviceaccount
tmpfs                    14.7G         0     14.7G   0% /proc/kcore
tmpfs                    14.7G         0     14.7G   0% /proc/timer_list
tmpfs                    14.7G         0     14.7G   0% /sys/firmware

Note that in the second case, my user is postgres:x:70:70::/var/lib/postgresql:/bin/sh, and I use the same ID (70) in the first case, so the numeric user ID should be identical , hence i dont see any reason why this might be a permissions error (although I could be wrong).

Some follow up questions, in case the main question of 'where's my /data contents?' isn't easy to answer:

1) Is there an easy way to confirm or download dynamically provisioned volume contents ? I'd rather not rely on kubernetes to debug wether a volume had the right contents, since container mounts can do weird things.

2) Is there any reason why a particular subdirectories contents arent visible in a new pod that is using the same PVC that I wrote the contents to, originally?

-- jayunit100
gce-persistent-disk
google-kubernetes-engine
kubernetes
kubernetes-pvc

Similar Questions

kubernetes HA joing work node error - cluster CA found in cluster-info configmap is invalid: public key
How to setup an audit policy into kube-apiserver?
Configure kubernetes to pull from a registry service
Dynamic storage provisioning on single bare metal node

2 Answers

4/12/2018

There is something strange with your mounts:

/ $ df -h | grep postgres
/dev/sdb                975.9M      2.5M    906.2M   0% /var/lib/postgresql
/dev/sda1                94.3G     31.0G     63.2G  33% /var/lib/postgresql/data

You are mounting /dev/sda1 which should be your container's root filesystem on top of the device /deb/sdb.

Just wondering, are you unmounting the volume /var/lib/postgresql/ before killing the first container?

-- Jordi Miralles
Source: StackOverflow
4/6/2018

I see that you have 2 mounts in your container:

$ df -h | grep postgres
/dev/sdb                975.9M      2.5M    906.2M   0% /var/lib/postgresql
/dev/sda1                94.3G     31.0G     63.2G  33% /var/lib/postgresql/data

Based on your Pod spec, Kubernetes is only handling the first mount, /var/lib/postgresql. Something else seems to be mounting this data directory, and it looks like it is mounting it from the container filesystem (/dev/sda1), instead of the GCE PD (/dev/sdb).

So your second Pod won't see any contents in the data directory because it's only mounted locally in the 1st container.

-- Michelle
Source: StackOverflow