kubernetes timescaledb statefulset: Changes lost on pod recreation

7/22/2019

I have a Timescaledb server running as StatefulSet in AKS. It appears when I delete and recreate timescaledb pod, the changes are lost even though the pod is associated to the initially associated PV (persistent volume). Any help is appreciated.

Below is the PV, PVC config of statefulset extracted by running kubectl get statefulset timescaledb -o yaml

  template:
    metadata:
      creationTimestamp: null
      labels:
        app: timescaledb
    spec:
      containers:
      - args:
        - -c
        - config_file=/etc/postgresql/postgresql.conf
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: timescaledb-secret
        image: docker.io/timescale/timescaledb:latest-pg9.6
        name: timescaledb-backend
        ports:
        - containerPort: 5432
          name: server
          protocol: TCP
        resources:
          requests:
            cpu: "3"
            memory: 6Gi
        volumeMounts:
        - mountPath: /var/lib/postgresql
          name: timescaledbdata
        - mountPath: /etc/postgresql
          name: timescaledb-config
      volumes:
      - configMap:
          defaultMode: 420
          name: timescaledb-config
        name: timescaledb-config
  volumeClaimTemplates:
  - metadata:
      annotations:
        volume.alpha.kubernetes.io/storage-class: standard
      creationTimestamp: null
      name: timescaledbdata
    spec:
      accessModes:
      - ReadWriteOnce
      dataSource: null
      resources:
        requests:
          storage: 200Gi
    status:
      phase: Pending

Below demonstrates a temp DB test_db created is lost after pod recreation and during the whole process, the pod was associated to the same PV/disk on Azure.

root@e70a91715239:~/keys# k get pvc -l app=timescaledb
NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
timescaledbdata-timescaledb-0   Bound    pvc-c7eb99cf-6a6b-11e9-b661-be660567cc75   200Gi      RWO            default        83d

root@e70a91715239:~/keys# k exec -ti timescaledb-0 bash
bash-4.4# psql -U postgres;
psql (9.6.13)
Type "help" for help.

postgres=# create database test_db;
CREATE DATABASE
postgres=# \l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
-----------+----------+----------+------------+------------+-----------------------
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 test_db   | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
(4 rows)


root@e70a91715239:~/keys# k get pods | grep timescale
timescaledb-0                         1/1     Running   0          12m
root@e70a91715239:~/keys# k delete pod/timescaledb-0                            
pod "timescaledb-0" deleted                                                                                                                                         
root@e70a91715239:~/keys# k get pods | grep timescale       
timescaledb-0                         1/1     Running   0          14s   

root@e70a91715239:~/keys# k exec -ti timescaledb-0 bash                                                                                                             
bash-4.4# psql -U postgres
psql (9.6.13)
Type "help" for help.

postgres=# \l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges                                                                                    
-----------+----------+----------+------------+------------+-----------------------                                                                                 
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +                                                                                 
           |          |          |            |            | postgres=CTc/postgres                                                                                  
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +                                                                                 
           |          |          |            |            | postgres=CTc/postgres                                                                                  
(3 rows)

root@e70a91715239:~/keys# k get pvc -l app=timescaledb
NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
timescaledbdata-timescaledb-0   Bound    pvc-c7eb99cf-6a6b-11e9-b661-be660567cc75   200Gi      RWO            default        83d

Possibly it is reinitialising as hinted. Please see logs. Any pointers on why it would do that.

Update 1: I had a look at the mounts in the timescale pod, it seems to have different partitions for /var/lib/postgresql and /var/lib/postgresql/data. I do not understand why.

Filesystem                Size      Used Available Use% Mounted on
overlay                  96.9G     22.1G     74.8G  23% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                     7.8G         0      7.8G   0% /sys/fs/cgroup
/dev/sda1                96.9G     22.1G     74.8G  23% /docker-entrypoint-initdb.d
/dev/sda1                96.9G     22.1G     74.8G  23% /dev/termination-log
shm                      64.0M      4.0K     64.0M   0% /dev/shm
/dev/sda1                96.9G     22.1G     74.8G  23% /etc/resolv.conf
/dev/sda1                96.9G     22.1G     74.8G  23% /etc/hostname
/dev/sda1                96.9G     22.1G     74.8G  23% /etc/hosts
/dev/sdc                196.7G     59.3M    196.7G   0% /var/lib/postgresql
/dev/sda1                96.9G     22.1G     74.8G  23% /var/lib/postgresql/data 

Do not understand how the above mounts happen for below config

        volumeMounts:
        - mountPath: /var/lib/postgresql
          name: timescaledbdata
        - mountPath: /etc/postgresql
          name: timescaledb-config
-- Rag
azure
azure-kubernetes
persistent-volumes
postgresql
timescaledb

1 Answer

7/26/2019

The problem was there was a VOLUME declaration for /var/lib/postgresql/data in postgres:9.6 Dockerfile which caused additional mount on the container. That mount was ephemeral when we had the volume mount at /var/lib/postgresql. But we were not able to mount AKS volume to /var/lib/postgresql/data because the volume came with lost+found subdirectory and Postgres expects empty directory to store DB files.

The fix was to mount volume at /var/lib/postgresql/data and tell Postgres to use a subdirectory underneath /var/lib/postgresql/data to store files with PGDATA env var.

Below are the relevant parts of fix in the k8s statefulset configuration

env:
- name: PGDATA
  value: "/var/lib/postgresql/data/dbfiles"        
...
volumeMounts:
- mountPath: /var/lib/postgresql/data
  name: timescaledata
-- Rag
Source: StackOverflow