PostgreSQL + Kubernetes: Role doesn't exist

1/15/2022

I used the following yaml to create a postgres deployment in my kubernetes cluser.

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: database-secret
  namespace: todo-app
data:
  # todoappdb
  db_name: dG9kb2FwcGRiCg==
  # todo_db_user
  username: dG9kb19kYl91c2VyCg==
  # password
  password: cGFzc3dvcmQK
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: database
  namespace: todo-app
  labels:
    app: database
spec:
  replicas: 1
  selector:
    matchLabels:
      app: database
  template:
    metadata:
      labels:
        app: database
    spec:
      containers:
        - name: database
          image: postgres:11
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: database-secret
                  key: password
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: database-secret
                  key: username
            - name: POSTGRES_DB
              valueFrom:
                secretKeyRef:
                  name: database-secret
                  key: db_name
---
apiVersion: v1
kind: Service
metadata:
  name: database
  namespace: todo-app
  labels:
    app: database
spec:
  type: NodePort
  selector:
    app: database
  ports:
    - port: 5432

When I try to run psql in the pod itself using the following command.

kubectl exec -it database-5764d75d58-msf7h  -n todo-app -- psql -U todo_db_user -d todoappdb

I get the following error.

psql: FATAL:  role "todo_db_user" does not exist

Here are the logs of the pod.

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data/pgdata -l logfile start


WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
waiting for server to start....2022-01-15 12:46:26.009 UTC [49] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-01-15 12:46:26.015 UTC [50] LOG:  database system was shut down at 2022-01-15 12:46:25 UTC
2022-01-15 12:46:26.017 UTC [49] LOG:  database system is ready to accept connections
 done
server started
CREATE DATABASE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

waiting for server to shut down...2022-01-15 12:46:26.369 UTC [49] LOG:  received fast shutdown request
.2022-01-15 12:46:26.369 UTC [49] LOG:  aborting any active transactions
2022-01-15 12:46:26.370 UTC [49] LOG:  background worker "logical replication launcher" (PID 56) exited with exit code 1
2022-01-15 12:46:26.371 UTC [51] LOG:  shutting down
2022-01-15 12:46:26.376 UTC [49] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2022-01-15 12:46:26.482 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2022-01-15 12:46:26.482 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2022-01-15 12:46:26.483 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-01-15 12:46:26.489 UTC [77] LOG:  database system was shut down at 2022-01-15 12:46:26 UTC
2022-01-15 12:46:26.492 UTC [1] LOG:  database system is ready to accept connections

Is there something wrong with the config?

When I don't use POSTGRES_USER env var, it works using role postgres. Also, with the current config I tried to use psql with the postgres role but that doesn't work either.

-- Nishant Mittal
docker
kubernetes
postgresql

1 Answer

1/15/2022

You have an error in your Secret. If you base64-decode these values:

data:
  # todoappdb
  db_name: dG9kb2FwcGRiCg==
  # todo_db_user
  username: dG9kb19kYl91c2VyCg==
  # password
  password: cGFzc3dvcmQK

You will find that they all include a terminal \n character:

$ kubectl get secret database-secret -o json > secret.json
$ jq '.data.username|@base64d' secret.json
"todo_db_user\n"
$ jq '.data.password|@base64d' secret.json
"password\n"
$ jq '.data.db_name|@base64d' secret.json
"todoappdb\n"

I suspect this is because you generate the values by running something like:

$ echo password | base64

But of course, the echo command emits a trailing newline (\n).

There are two ways of solving this:

  1. Use stringData instead of data in your Secret so you can just write the unencoded values:

    apiVersion: v1
    kind: Secret
    type: Opaque
    metadata:
      name: database-secret
    stringData:
      db_name: todoappdb
      username: todo_db_user
      password: password
    
  2. Instruct echo to not emit a trailing newline:

    $ echo -n todo_db_user | base64

    (Or use something like printf which doesn't emit a newline by default).

I would opt for the first option (using stringData) because it's much simpler.

-- larsks
Source: StackOverflow