I have a Timescaledb server running as StatefulSet in AKS. It appears when I delete and recreate timescaledb pod, the changes are lost even though the pod is associated to the initially associated PV (persistent volume). Any help is appreciated.
Below is the PV, PVC config of statefulset extracted by running kubectl get statefulset timescaledb -o yaml
template:
metadata:
creationTimestamp: null
labels:
app: timescaledb
spec:
containers:
- args:
- -c
- config_file=/etc/postgresql/postgresql.conf
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: timescaledb-secret
image: docker.io/timescale/timescaledb:latest-pg9.6
name: timescaledb-backend
ports:
- containerPort: 5432
name: server
protocol: TCP
resources:
requests:
cpu: "3"
memory: 6Gi
volumeMounts:
- mountPath: /var/lib/postgresql
name: timescaledbdata
- mountPath: /etc/postgresql
name: timescaledb-config
volumes:
- configMap:
defaultMode: 420
name: timescaledb-config
name: timescaledb-config
volumeClaimTemplates:
- metadata:
annotations:
volume.alpha.kubernetes.io/storage-class: standard
creationTimestamp: null
name: timescaledbdata
spec:
accessModes:
- ReadWriteOnce
dataSource: null
resources:
requests:
storage: 200Gi
status:
phase: PendingBelow demonstrates a temp DB test_db created is lost after pod recreation and during the whole process, the pod was associated to the same PV/disk on Azure.
root@e70a91715239:~/keys# k get pvc -l app=timescaledb
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
timescaledbdata-timescaledb-0 Bound pvc-c7eb99cf-6a6b-11e9-b661-be660567cc75 200Gi RWO default 83d
root@e70a91715239:~/keys# k exec -ti timescaledb-0 bash
bash-4.4# psql -U postgres;
psql (9.6.13)
Type "help" for help.
postgres=# create database test_db;
CREATE DATABASE
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
test_db | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
(4 rows)
root@e70a91715239:~/keys# k get pods | grep timescale
timescaledb-0 1/1 Running 0 12m
root@e70a91715239:~/keys# k delete pod/timescaledb-0
pod "timescaledb-0" deleted
root@e70a91715239:~/keys# k get pods | grep timescale
timescaledb-0 1/1 Running 0 14s
root@e70a91715239:~/keys# k exec -ti timescaledb-0 bash
bash-4.4# psql -U postgres
psql (9.6.13)
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
root@e70a91715239:~/keys# k get pvc -l app=timescaledb
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
timescaledbdata-timescaledb-0 Bound pvc-c7eb99cf-6a6b-11e9-b661-be660567cc75 200Gi RWO default 83dPossibly it is reinitialising as hinted. Please see logs. Any pointers on why it would do that.
Update 1: I had a look at the mounts in the timescale pod, it seems to have different partitions for /var/lib/postgresql and /var/lib/postgresql/data. I do not understand why.
Filesystem Size Used Available Use% Mounted on
overlay 96.9G 22.1G 74.8G 23% /
tmpfs 64.0M 0 64.0M 0% /dev
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/sda1 96.9G 22.1G 74.8G 23% /docker-entrypoint-initdb.d
/dev/sda1 96.9G 22.1G 74.8G 23% /dev/termination-log
shm 64.0M 4.0K 64.0M 0% /dev/shm
/dev/sda1 96.9G 22.1G 74.8G 23% /etc/resolv.conf
/dev/sda1 96.9G 22.1G 74.8G 23% /etc/hostname
/dev/sda1 96.9G 22.1G 74.8G 23% /etc/hosts
/dev/sdc 196.7G 59.3M 196.7G 0% /var/lib/postgresql
/dev/sda1 96.9G 22.1G 74.8G 23% /var/lib/postgresql/data Do not understand how the above mounts happen for below config
volumeMounts:
- mountPath: /var/lib/postgresql
name: timescaledbdata
- mountPath: /etc/postgresql
name: timescaledb-configThe problem was there was a VOLUME declaration for /var/lib/postgresql/data in postgres:9.6 Dockerfile which caused additional mount on the container. That mount was ephemeral when we had the volume mount at /var/lib/postgresql. But we were not able to mount AKS volume to /var/lib/postgresql/data because the volume came with lost+found subdirectory and Postgres expects empty directory to store DB files.
The fix was to mount volume at /var/lib/postgresql/data and tell Postgres to use a subdirectory underneath /var/lib/postgresql/data to store files with PGDATA env var.
Below are the relevant parts of fix in the k8s statefulset configuration
env:
- name: PGDATA
value: "/var/lib/postgresql/data/dbfiles"
...
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: timescaledata