I'm attempting to create a persistent volume claim for my pgadmin deployment so I can persist my settings, servers, etc. when I rollout updates after each deployment from CD pipeline.
In my logs I'm getting the following errors:
...
[2020-10-05 00:54:56 +0000] [91] [INFO] Worker exiting (pid: 91)
WARNING: Failed to set ACL on the directory containing the configuration database:
[Errno 1] Operation not permitted: '/var/lib/pgadmin'
HINT : You may need to manually set the permissions on
/var/lib/pgadmin to allow pgadmin to write to it.
ERROR : Failed to create the directory /var/lib/pgadmin/sessions:
[Errno 13] Permission denied: '/var/lib/pgadmin/sessions'
HINT : Create the directory /var/lib/pgadmin/sessions, ensure it is writeable by
'pgadmin', and try again, or, create a config_local.py file
and override the SESSION_DB_PATH setting per
https://www.pgadmin.org/docs/pgadmin4/4.26/config_py.htmlJust a bunch of permission failures for writing:
apiVersion: apps/v1
kind: Deployment
metadata:
name: pgadmin
spec:
selector:
matchLabels:
app: pgadmin
replicas: 1
template:
metadata:
labels:
app: pgadmin
spec:
containers:
- name: pgadmin4
image: dpage/pgadmin4
volumeMounts:
- mountPath: /var/lib/pgadmin
name: pgadminstorage
env:
- name: PGADMIN_DEFAULT_EMAIL
valueFrom:
secretKeyRef:
name: un
key: un
- name: PGADMIN_DEFAULT_PASSWORD
valueFrom:
secretKeyRef:
name: pw
key: pw
- name: PGADMIN_PORT
value: "80"
ports:
- containerPort: 80
name: pgadminport
volumes:
- name: pgadminstorage
persistentVolumeClaim:
claimName: pgadmin-persistent-volume-claims-cfgapiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pgadmin-persistent-volume-claims-cfg
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1GiWhat could I be missing here?
This might be a problem specific to digitalocean and inability to set permissions. PVC will set perms to root, but writing as pgadmin is causing issues at startup Adding this to my pgadmin deployment fixed everything
initContainers:
- name: pgadmin-data-permission-fix
image: busybox
command: ["/bin/chown", "-R", "5050:5050", "/var/lib/pgadmin"]
volumeMounts:
- name: pgadminstorage
mountPath: /var/lib/pgadminYou could probably also chmod recursive on the dir as well and also be fine.
I've replicated your issue. Root cause is with PgAdmin issue, not Kubernetes. Pods will be deployed without issue. You will receive error as container won't be able to create folder inside folder /var/lib. If you will check pgadmin pod logs - kubectl logs <pgadmin-pod> you will see errors like:
$ kubectl logs pgadmin-d569b67fd-8rnkc
WARNING: Failed to set ACL on the directory containing the configuration database:
[Errno 1] Operation not permitted: '/var/lib/pgadmin'
HINT : You may need to manually set the permissions on
/var/lib/pgadmin to allow pgadmin to write to it.
ERROR : Failed to create the directory /var/lib/pgadmin/sessions:
[Errno 13] Permission denied: '/var/lib/pgadmin/sessions'
HINT : Create the directory /var/lib/pgadmin/sessions, ensure it is writeable by
'pgadmin', and try again, or, create a config_local.py file
and override the SESSION_DB_PATH setting per
https://www.pgadmin.org/docs/pgadmin4/4.26/config_py.html
sudo: setrlimit(RLIMIT_CORE): Operation not permittedIf you will check /var/lib/ folder permissions you will se that you can only Read and Execute, so you won't be able to create in this folder anything (as default, you will be logged as pgadmin user).
drwxr-xr-x 1 root root 4096 Sep 5 14:01 libDepends on your needs you can resolve it in a few ways. As fastest workaround you can just change path to folder which allows Write, like tmp.
drwxrwxrwt 1 root root 4096 Oct 5 14:28 tmpIn YAML it would looks like:
containers:
- name: pgadmin4
image: dpage/pgadmin4
volumeMounts:
- mountPath: /var/tmp/pgadmin
name: pgadminstorageWhen you will check logs, there won't be any issues.
$ kubectl logs pgadmin-6bb74cffb8-6q9tr
NOTE: Configuring authentication for SERVER mode.
sudo: setrlimit(RLIMIT_CORE): Operation not permitted
[2020-10-05 14:28:15 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2020-10-05 14:28:15 +0000] [1] [INFO] Listening at: http://[::]:80 (1)
[2020-10-05 14:28:15 +0000] [1] [INFO] Using worker: threads
/usr/local/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
return io.open(fd, *args, **kwargs)
[2020-10-05 14:28:15 +0000] [89] [INFO] Booting worker with pid: 89
user@cloudshell:~/pgadmin (project)$
Regarding PgAdmin permissions issue there was already a few topics on StackOverflow or Github like:
OSError: [Errno 13] Permission denied: '/var/lib/pgadmin'
pgadmin exit code 3 PermissionError: [Errno 13] Permission denied: '/var/lib/pgadmin/sessions'
[stable/pgadmin] files in /var/lib/pgadmin/sessions crash the pod
In short, you could try to manually change permissions or use specific user.
In addition, if you are using Cloud environment, you could think about using CloudSQL, instead of trying to put database into cloud. For example PostgreSQL with GKE
EDIT
As per @Ryan comment below this answer, you can also use Init Containers to change /var/lib/ permissions. Each init container must complete successfully before the next one starts and it runs before app containers in a pod.
specialized containers that run before app containers in a Pod. Init containers can contain utilities or setup scripts not present in an app image.