minikube postgres statefulset error - FATAL: role "password=" does not exist

8/25/2019

I'm using the Kubernetes Postgres StateulSet example repo here by Arianit Uka: https://github.com/arianitu/postgres-statefulset

with minikube and my pod container isn't coming up. The secrets have been applied and it seems to have the POSTGRES_PASSWORD env var just fine. There's no password= in the repo's code so I'm lost where the issue may be. Checking the logs gives this:

> $ k logs postgres-0 --all-containers=true                                                                                              ⬡ 8.11.4 [±hauser ●●]
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data/pgdata -l logfile start

waiting for server to start....2019-08-25 07:22:31.295 UTC [41] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-08-25 07:22:31.307 UTC [42] LOG:  database system was shut down at 2019-08-25 07:22:31 UTC
2019-08-25 07:22:31.310 UTC [41] LOG:  database system is ready to accept connections
 done
server started

/usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d/create-dev-db.sh
CREATE DATABASE
GRANT

/usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d/create-replica-user.sh
CREATE ROLE

2019-08-25 07:22:31.751 UTC [41] LOG:  received fast shutdown request
waiting for server to shut down....2019-08-25 07:22:31.752 UTC [41] LOG:  aborting any active transactions
2019-08-25 07:22:31.753 UTC [41] LOG:  worker process: logical replication launcher (PID 48) exited with exit code 1
2019-08-25 07:22:31.755 UTC [43] LOG:  shutting down
2019-08-25 07:22:31.774 UTC [41] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2019-08-25 07:22:31.862 GMT [1] LOG:  skipping missing configuration file "/etc/replica.conf"
2019-08-25 07:22:31.862 GMT [1] LOG:  skipping missing configuration file "/etc/replica.conf"
2019-08-25 07:22:31.865 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2019-08-25 07:22:31.865 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2019-08-25 07:22:31.869 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-08-25 07:22:31.888 UTC [68] LOG:  database system was shut down at 2019-08-25 07:22:31 UTC
2019-08-25 07:22:31.893 UTC [1] LOG:  database system is ready to accept connections
2019-08-25 07:23:20.352 UTC [75] FATAL:  role "password=" does not exist
2019-08-25 07:23:23.595 UTC [82] FATAL:  role "password=" does not exist

Here's what the minikube dashboard looks like:

minikube dashbaord


@Rahman, I took out the password secrets and am using just plain values for now in both statefulset-master.yml and statefulset-replica.yml:

          env:
            ...

            - name: POSTGRES_PASSWORD
              value: master-password

            - name: REPLICATION_PASSWORD
              value: replica-password

I've also commented out the replica section in service.yml as I'm only interesting in running a single pod/database at the moment:


service.yml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: postgres
  name: postgres
spec:
  type: ClusterIP
  ports:
  - name: postgres
    port: 5432
    protocol: TCP
    targetPort: 5432
  selector:
    app: postgres
    
# ---

# apiVersion: v1
# kind: Service
# metadata:
#   labels:
#     app: postgres-replica
#   name: postgres-replica

# spec:
#   type: ClusterIP
#   ports:
#   - name: postgres-replica
#     port: 5432
#     protocol: TCP
#     targetPort: 5432
#   selector:
#     app: postgres-replica
    
# ---

statefulset-master.yml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  updateStrategy:
    type: RollingUpdate
  
  selector:
    matchLabels:
      app: postgres

  serviceName: postgres
  replicas: 1
  template:
    metadata:
      labels:
        app: postgres 
    spec:
      volumes:
        - name: postgres-config
          configMap:
            name: postgres

        # - name: shared
        #   emptyDir: {}
            
      terminationGracePeriodSeconds: 10

      containers:
        - name: postgres
          image: postgres:10.5
          args: ['-c', 'config_file=/etc/postgres.conf', '-c', 'hba_file=/etc/pg_hba.conf']
          
          imagePullPolicy: IfNotPresent
        
          ports:
            - name: postgres
              containerPort: 5432
              protocol: TCP
          
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
          
          env:
            - name: POSTGRES_USER
              value: postgres
          
            - name: PGUSER
              value: postgres
          
            - name: POSTGRES_DB
              value: postgres
            
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata

            - name: POSTGRES_PASSWORD
              value: master-password
              # valueFrom:
              #   secretKeyRef:
              #     key: password
              #     name: postgres
                  
            - name: REPLICATION_PASSWORD
              value: replica-password
              # valueFrom:
              #   secretKeyRef:
              #     key: replicaPassword
              #     name: postgres
              
            - name: POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
              
          livenessProbe:
            exec:
              command:
                - sh
                - -c
                - exec pg_isready --host $POD_IP
            failureThreshold: 6
            initialDelaySeconds: 60
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5

          readinessProbe:
            exec:
              command:
                - sh
                - -c
                - exec pg_isready --host $POD_IP
            failureThreshold: 3
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 3

          volumeMounts:
            - mountPath: /var/lib/postgresql/data/pgdata
              name: postgres
              subPath: postgres-db
              
            - name: postgres-config
              mountPath: /etc/postgres.conf
              subPath: postgres.conf
              
            - name: postgres-config
              mountPath: /etc/master.conf
              subPath: master.conf
              
            - name: postgres-config
              mountPath: /etc/pg_hba.conf
              subPath: pg_hba.conf
              
            - name: postgres-config
              mountPath: /docker-entrypoint-initdb.d/create-replica-user.sh
              subPath: create-replica-user.sh

            - name: postgres-config
              mountPath: /docker-entrypoint-initdb.d/create-dev-db.sh
              subPath: create-dev-db.sh

            # - name: shared
            #   mountPath: /User/Shared

        - name: hauser
          image: mikeumus/hauser
          # volumeMounts:
          #   - name: shared
          #     mountPath: /User/Shared
        
  volumeClaimTemplates:
  - metadata:
      name: postgres
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "standard"
      resources:
        requests:
          storage: 3Gi

statefulset-replica.yml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-replica
spec:
  updateStrategy:
    type: RollingUpdate
  
  selector:
    matchLabels:
      app: postgres-replica

  serviceName: postgres-replica
  replicas: 1
  template:
    metadata:
      labels:
        app: postgres-replica 
    spec:
      volumes:
        - name: postgres-config
          configMap:
            name: postgres
            
      terminationGracePeriodSeconds: 10
      
      initContainers:
        - name: setup-replica-data-directory
          image: postgres:10.5
          
          env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  key: replicaPassword
                  name: postgres

          command:
          - sh
          - -c
          - |
            if [ -z "$(ls -A /var/lib/postgresql/data/pgdata)" ]; then
                echo "Running pg_basebackup to catch up replication server...";
                pg_basebackup -R -h postgres -D /var/lib/postgresql/data/pgdata -P -U replication; 
                chown -R postgres:postgres $PGDATA;
            else
                echo "Skipping pg_basebackup because directory is not empty"; 
            fi

          volumeMounts:
            - mountPath: /var/lib/postgresql/data/pgdata
              name: postgres-replica
              subPath: postgres-db

      containers:
        - name: postgres-replica
          image: postgres:10.5
          args: ['-c', 'config_file=/etc/postgres.conf']
          
          imagePullPolicy: IfNotPresent
        
          ports:
            - name: postgres-rep
              containerPort: 5432
              protocol: TCP
          
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
          
          env:
            - name: POSTGRES_USER
              value: postgres
          
            - name: PGUSER
              value: postgres
          
            - name: POSTGRES_DB
              value: postgres
            
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          
            - name: POSTGRES_PASSWORD
              value: master-password
              # valueFrom:
              #   secretKeyRef:
              #     key: password
              #     name: postgres
                  
            - name: REPLICATION_PASSWORD
              value: replica-password
              # valueFrom:
              #   secretKeyRef:
              #     key: replicaPassword
              #     name: postgres
              
            - name: POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
              
          livenessProbe:
            exec:
              command:
                - sh
                - -c
                - exec pg_isready --host $POD_IP
            failureThreshold: 6
            initialDelaySeconds: 60
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5

          readinessProbe:
            exec:
              command:
                - sh
                - -c
                - exec pg_isready --host $POD_IP
            failureThreshold: 3
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 3

          volumeMounts:
            - mountPath: /var/lib/postgresql/data/pgdata
              name: postgres-replica
              subPath: postgres-db
            
            - name: postgres-config
              mountPath: /etc/postgres.conf
              subPath: postgres.conf

            - name: postgres-config
              mountPath: /etc/replica.conf
              subPath: replica.conf
             
  volumeClaimTemplates:
  - metadata:
      name: postgres-replica
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "standard"
      resources:
        requests:
          storage: 3Gi

config/create_configmap.sh

kubectl create configmap postgres --from-file=postgres.conf --from-file=master.conf --from-file=replica.conf --from-file=pg_hba.conf --from-file=create-replica-user.sh --from-file=create-dev-db.sh

config/create-replica-user.sh

#!/bin/bash
set -e

psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
    CREATE ROLE replication WITH REPLICATION PASSWORD '$REPLICATION_PASSWORD' LOGIN
EOSQL

Here's the order I start/apply resources:

  1. minikube start
  2. cd config I get some missing error unless I'm in config dir when I do this
  3. . ./create_configmap.sh
  4. cd ..
  5. k apply -f ./config/secret.yml
  6. k apply -f ./service.yml
  7. k apply -f ./statefulset-master.yml

Link to my fork of postgres-statefulset repo: https://gitlab.com/mikeumus/postgres-statefulset

-- Mikeumus
docker
kubernetes
kubernetes-statefulset
minikube
postgresql

1 Answer

8/26/2019

It wasn't the postgres-stateful code that was the problem. The "password=" code was in the Hauser code I added as another container to statefulset-master.yml

-- Mikeumus
Source: StackOverflow