K
Q

The server must be started by the user that owns the data directory

7/25/2019

I am trying to get some persistant storage for a docker instance of PostgreSQL running on Kubernetes. However, the pod fails with

FATAL:  data directory "/var/lib/postgresql/data" has wrong ownership
HINT:  The server must be started by the user that owns the data directory.

This is the NFS configuration:

% exportfs -v
/srv/nfs/postgresql/postgres-registry
        kubehost*.example.com(rw,wdelay,insecure,no_root_squash,no_subtree_check,sec=sys,rw,no_root_squash,no_all_squash)
$ ls -ldn /srv/nfs/postgresql/postgres-registry
drwxrwxrwx. 3 999 999 4096 Jul 24 15:02 /srv/nfs/postgresql/postgres-registry
$ ls -ln /srv/nfs/postgresql/postgres-registry
total 4
drwx------. 2 999 999 4096 Jul 25 08:36 pgdata

The full log from the pod:

2019-07-25T07:32:50.617532000Z The files belonging to this database system will be owned by user "postgres".
2019-07-25T07:32:50.618113000Z This user must also own the server process.
2019-07-25T07:32:50.619048000Z The database cluster will be initialized with locale "en_US.utf8".
2019-07-25T07:32:50.619496000Z The default database encoding has accordingly been set to "UTF8".
2019-07-25T07:32:50.619943000Z The default text search configuration will be set to "english".
2019-07-25T07:32:50.620826000Z Data page checksums are disabled.
2019-07-25T07:32:50.621697000Z fixing permissions on existing directory /var/lib/postgresql/data ... ok
2019-07-25T07:32:50.647445000Z creating subdirectories ... ok
2019-07-25T07:32:50.765065000Z selecting default max_connections ... 20
2019-07-25T07:32:51.035710000Z selecting default shared_buffers ... 400kB
2019-07-25T07:32:51.062039000Z selecting default timezone ... Etc/UTC
2019-07-25T07:32:51.062828000Z selecting dynamic shared memory implementation ... posix
2019-07-25T07:32:51.218995000Z creating configuration files ... ok
2019-07-25T07:32:51.252788000Z 2019-07-25 07:32:51.251 UTC [79] FATAL:  data directory "/var/lib/postgresql/data" has wrong ownership
2019-07-25T07:32:51.253339000Z 2019-07-25 07:32:51.251 UTC [79] HINT:  The server must be started by the user that owns the data directory.
2019-07-25T07:32:51.262238000Z child process exited with exit code 1
2019-07-25T07:32:51.263194000Z initdb: removing contents of data directory "/var/lib/postgresql/data"
2019-07-25T07:32:51.380205000Z running bootstrap script ...

The deployment has the following in:

  securityContext:
    runAsUser: 999
    supplementalGroups: [999,1000]
    fsGroup: 999

What am I doing wrong?

Edit: Added storage.yaml file:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: postgres-registry-pv-volume
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 192.168.3.7
    path: /srv/nfs/postgresql/postgres-registry
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: postgres-registry-pv-claim
  labels:
    app: postgres-registry
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi

Edit: And the full deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: postgres-registry
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: postgres-registry
    spec:
      securityContext:
        runAsUser: 999
        supplementalGroups: [999,1000]
        fsGroup: 999
      containers:
        - name: postgres-registry
          image: postgres:latest
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432
          env:
              - name: POSTGRES_DB
                value: postgresdb
              - name: POSTGRES_USER
                value: postgres
              - name: POSTGRES_PASSWORD
                value: Sekret
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              subPath: "pgdata"
              name: postgredb-registry-persistent-storage
      volumes:
        - name: postgredb-registry-persistent-storage
          persistentVolumeClaim:
            claimName: postgres-registry-pv-claim

Even more debugging adding:

command: ["/bin/bash", "-c"]
args:["id -u; ls -ldn /var/lib/postgresql/data"]

Which returned:

999
drwx------. 2 99 99 4096 Jul 25 09:11 /var/lib/postgresql/data

Clearly, the UID/GID are wrong. Why?

Even with the work around suggested by Jakub Bujny, I get this:

 2019-07-25T09:32:08.734807000Z The files belonging to this database system will be owned by user "postgres".
2019-07-25T09:32:08.735335000Z This user must also own the server process.
2019-07-25T09:32:08.736976000Z The database cluster will be initialized with locale "en_US.utf8".
2019-07-25T09:32:08.737416000Z The default database encoding has accordingly been set to "UTF8".
2019-07-25T09:32:08.737882000Z The default text search configuration will be set to "english".
2019-07-25T09:32:08.738754000Z Data page checksums are disabled.
2019-07-25T09:32:08.739648000Z fixing permissions on existing directory /var/lib/postgresql/data ... ok
2019-07-25T09:32:08.766606000Z creating subdirectories ... ok
2019-07-25T09:32:08.852381000Z selecting default max_connections ... 20
2019-07-25T09:32:09.119031000Z selecting default shared_buffers ... 400kB
2019-07-25T09:32:09.145069000Z selecting default timezone ... Etc/UTC
2019-07-25T09:32:09.145730000Z selecting dynamic shared memory implementation ... posix
2019-07-25T09:32:09.168161000Z creating configuration files ... ok
2019-07-25T09:32:09.200134000Z 2019-07-25 09:32:09.199 UTC [70] FATAL:  data directory "/var/lib/postgresql/data" has wrong ownership
2019-07-25T09:32:09.200715000Z 2019-07-25 09:32:09.199 UTC [70] HINT:  The server must be started by the user that owns the data directory.
2019-07-25T09:32:09.208849000Z child process exited with exit code 1
2019-07-25T09:32:09.209316000Z initdb: removing contents of data directory "/var/lib/postgresql/data"
2019-07-25T09:32:09.274741000Z running bootstrap script ... 999
2019-07-25T09:32:09.278124000Z drwx------. 2 99 99 4096 Jul 25 09:32 /var/lib/postgresql/data
-- Sardathrion - against SE abuse
docker
kubernetes
nfs
postgresql

Similar Questions

K8s Service as DaemonSet
kubernetes - can we create 2 node master-only cluster with High availability
How to authenticate to Cloud Function using a node.js client
How to log to Kubernetes Container Log from Python process
Forbidden error to connect to kubernetes cluster in gitlab CI
Cannot list out stopped container using docker ps -a
starting container process caused "process_linux.go:303: getting the final child's pid from pipe caused \"EOF\"": unknown

3 Answers

7/25/2019

Using your setup and ensuring the nfs mount is owned by 999:999 it worked just fine. You're also missing an 's' in your name: postgredb-registry-persistent-storage

And with your subPath: "pgdata" do you need to change the $PGDATA? I didn't include the subpath for this.

$ sudo mount 172.29.0.218:/test/nfs ./nfs

$ sudo su -c "ls -al ./nfs" postgres
total 8
drwx------ 2 postgres postgres 4096 Jul 25 14:44 .
drwxrwxr-x 3 rei      rei      4096 Jul 25 14:44 ..

$ kubectl apply -f nfspv.yaml 
persistentvolume/postgres-registry-pv-volume created
persistentvolumeclaim/postgres-registry-pv-claim created

$ kubectl apply -f postgres.yaml 
deployment.extensions/postgres-registry created

$ sudo su -c "ls -al ./nfs" postgres 
total 124
drwx------ 19 postgres postgres  4096 Jul 25 14:46 .
drwxrwxr-x  3 rei      rei       4096 Jul 25 14:44 ..
drwx------  3 postgres postgres  4096 Jul 25 14:46 base
drwx------  2 postgres postgres  4096 Jul 25 14:46 global
drwx------  2 postgres postgres  4096 Jul 25 14:46 pg_commit_ts
. . .

I noticed using nfs: directly in the persistent volume took significantly longer to initialize the database, whereas using hostPath: to the mounted nfs volume behaved normally.

So after a few minutes:

$ kubectl logs postgres-registry-675869694-9fp52 | tail -n 3
2019-07-25 21:50:57.181 UTC [30] LOG:  database system is ready to accept connections                                                             
 done                                                                                                                                             
server started

$ kubectl exec -it postgres-registry-675869694-9fp52 psql                                    
psql (11.4 (Debian 11.4-1.pgdg90+1))                                                                                                              
Type "help" for help.                                                                                                                             

postgres=#

Checking the uid/gid

$ kubectl exec -it postgres-registry-675869694-9fp52 bash
postgres@postgres-registry-675869694-9fp52:/$ whoami && id -u && id -g
postgres                                                                                                                                          
999                                                                                                                                               
999

nfspv.yaml:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: postgres-registry-pv-volume
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 172.29.0.218
    path: /test/nfs
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: postgres-registry-pv-claim
  labels:
    app: postgres-registry
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi

postgres.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: postgres-registry
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: postgres-registry
    spec:
      securityContext:
        runAsUser: 999
        supplementalGroups: [999,1000]
        fsGroup: 999
      containers:
        - name: postgres-registry
          image: postgres:latest
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432
          env:
              - name: POSTGRES_DB
                value: postgresdb
              - name: POSTGRES_USER
                value: postgres
              - name: POSTGRES_PASSWORD
                value: Sekret
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgresdb-registry-persistent-storage
      volumes:
        - name: postgresdb-registry-persistent-storage
          persistentVolumeClaim:
            claimName: postgres-registry-pv-claim
-- wgl
Source: StackOverflow
7/25/2019

I cannot explain why those 2 IDs are different but as workaround I would try to override postgres's entrypoint with

command: ["/bin/bash", "-c"]
args: ["chown -R 999:999 /var/lib/postgresql/data && ./docker-entrypoint.sh postgres"]
-- Jakub Bujny
Source: StackOverflow
9/4/2019

This type of errors is quite common when you link a NTFS directory into your docker container. NTFS directories don't support ext3 file & directory access control. The only way to make it work is to link directory from a ext3 drive into your container.

I got a bit desperate when I played around Apache / PHP containers with linking the www folder. After I linked files reside on a ext3 filesystem the problem disappear.

I published a short Docker tutorial on youtube, may it helps to understand this problem: https://www.youtube.com/watch?v=eS9O05TTFjM

-- Elmar Dott
Source: StackOverflow