K
Q

Invalid resource manager ID in primary checkpoint record

2/24/2022

I've update my Airbyte image from 0.35.2-alpha to 0.35.37-alpha. running in kubernetes

When the system rolled out the db pod wouldn't terminate and I a terrible mistake deleted the pod. When it came back up, I get an error - 

PostgreSQL Database directory appears to contain a database; Skipping initialization

2022-02-24 20:19:44.065 UTC [1] LOG:  starting PostgreSQL 13.6 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027, 64-bit
2022-02-24 20:19:44.065 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2022-02-24 20:19:44.065 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2022-02-24 20:19:44.071 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-02-24 20:19:44.079 UTC [21] LOG:  database system was shut down at 2022-02-24 20:12:55 UTC
2022-02-24 20:19:44.079 UTC [21] LOG:  invalid resource manager ID in primary checkpoint record
2022-02-24 20:19:44.079 UTC [21] PANIC:  could not locate a valid checkpoint record
2022-02-24 20:19:44.530 UTC [1] LOG:  startup process (PID 21) was terminated by signal 6: Aborted
2022-02-24 20:19:44.530 UTC [1] LOG:  aborting startup due to startup process failure
2022-02-24 20:19:44.566 UTC [1] LOG:  database system is shut down

Pretty sure the WAL file is corrupted, but I'm not sure how to fix this.

-- Jeremy
kubernetes
postgresql

Similar Questions

How can i change a service running on one port number, now i want to run another service on same old port number
Kubernetes on GKE how to write data on PVC with readonlymany mode
OpenShift - List serviceaccounts with privileged scc
environment variables in Docker images in Kubernetes Cluster
How to get annotation from namespace in side car injector

1 Answer

2/24/2022

Warning - there is a potential for data loss

This is a test system, so I wasn't concerned with keeping the latest transactions, and had no backup.

First I overrode the container command to keep the container running but not try to start postgres. 

...
    spec:
      containers:
        - name: airbyte-db-container
          image: airbyte/db
          command: ["sh"]
          args: ["-c", "while true; do echo $(date -u) >> /tmp/run.log; sleep 5; done"]
...

And spawned a shell on the pod - 

kubectl exec -it -n airbyte airbyte-db-xxxx -- sh

Run pg_reset_wal

# dry-run first
pg_resetwal --dry-run /var/lib/postgresql/data/pgdata

Success!

pg_resetwal /var/lib/postgresql/data/pgdata
Write-ahead log reset

Then removed the temp command in the container, and postgres started up correctly!

-- Jeremy
Source: StackOverflow