We are running Postgres in Kubernetes. Our Prometheus pod monitoring always shows that Postgres fills up the entire pods memory in a shared or cached state.

free -h
              total        used        free      shared  buff/cache   available
Mem:            62G        1.7G         13G         11G         47G         48G


ps -u postgres o pid= | sed 's#.*#/proc/&/smaps#' | sed 's/ //g#/proc/&/smaps#' | \
	xargs grep -s ^Pss: | awk '{A+=$2} END{print "PSS: " A}' 
PSS: 13220013 kb

ps -u postgres o pid,rss:8,cmd | awk 'NR>1 {A+=$2} {print} END{print "RSS: " A}' | tail -1
RSS: 38794236 kb

Correct me if I am wrong, but since the memory displayed in top and ps(RSS/RES) is shared memory, this means that Posgres isnt using that memory, its only reserved for when it needs it and other processes can also use that memory. Some articles say that one needs to cat /proc/<PID>/smaps and check the PSS to find the actual memory usage of Postgres.

We recently got OOM errors, but we where unable to pick it up in our monitoring. Due to the fact that our pod memory monitoring is always displaying 90% usage as its only monitoring the RSS/RES memory which is includes the shared/cached memory as well. So we didnt see any increase in RAM when the OOM errors happened and our database went down. The error was caused by a new query we introduced to our backend which used large amounts of memory per query.

We have prometheus-postgres-exporter installed giving us good Postgres metrics, but this didnt show us that we had queries using large amounts memory, but maybe our Grafana graph is missing that graph?

Because we are running Postgres in Kubernetes exporting the PSS memory is a hassle. So it feels like I am missing something.

So how does one monitor the memory usage of Postgres ? How does one pick up on queries using too much memory or pick up on postgres using too much memory due to load?