Session spike during database migration with multi-instance of a service with rolling upgrade strategy of Kubernetes

12/24/2019

I have multiple-instance of an application[1], the ORM tool used is Hibernate (hibernate.hbm2ddl.auto=update) that points to the same AWS RDS (Postgress) database.

When there is a manual change in DB schema like constraints drop, column drop and the application deployment is done rolling upgrade strategy of Kubernetes[2] the older pods are still around with old schema, whereas, due to update strategy of hibernate the schema has already changed and I start to see below spike in database session. Can someone provide lead where the issue could be?

SELECT *
FROM pg_stat_activity
order by query;

enter image description here

Background:
[1]: These are containerized application that follows the CICD regime, canary deployment and Kubernetes rolling upgrade.
[2]: When the rolling upgrade strategy of Kubernetes is used, for high availability Kubernetes upgrade pod by pod while old pods are running, if still unclear, read this article for more details.

-- Vishrant
amazon-rds
hibernate
kubernetes
postgresql
rolling-updates

0 Answers