How to solve liquibase waiting for changelog lock problem in several pods in OpenShift cluster?

4/23/2020

Please say something about this problem:

We are supporting several microservices written in Java using Spring Boot and deployed in OpenShift. Some microservices communacates with databases. We often run a single microservice in multiple pods in a single deployment. When each microservice starts, it starts liquibase, which tries to update the database. The problem is that sometimes one pod fails with waiting for the changelog lock issue. When for some reason this happens in production OpenShift cluster, we expect other pods to fail while restarting because of the same problem with changelog lock issue. So in the worst case scenario, all pods will wait for the lock to be lifted.

We want liquidbase automatically prepare our database schemas when pod is starting.

So please tell the best way to solve this problem? Is it good to store this logic in every microservice? How can we automatically solve the problem when the liquidbase changelog lock problem appears? Do we need to put the database preparation logic in a separate deployment?

Thanks

#Updated

So maybe I should paraphrase my question. What is the best way to run db migration in term of microservice architecture? Maybe we should not use db migration in each pod? Maybe it is better to do it with separate deployment or do it with some extra Jenkins job not in OpenShift at all?

-- AlexCrazy
database
kubernetes
liquibase
openshift
spring-boot

1 Answer

4/23/2020

When Liquibase kicks in during the spring-boot app deployment, it performs (on a very high level) the following steps:

  1. lock the database (create a record in databasechangeloglock)
  2. execute changeLogs;
  3. remove database lock;

So if you interrupt application deployment while Liquibase is between steps 1 and 3, then your database will remain locked. So when you'll try to redeploy your app, Liquibase will fail, because it will treat your database as locked.

So you have to unlock the database before deploying the app again.

There are two options that I'm aware of:

  1. Clear databasechangeloglock table or set locked to false. Which is DELETE FROM databasechangeloglock or UPDATE databasechangeloglock SET locked=0
  2. Execute liquibase releaseLocks command. You can find documentation about it here and here.
-- htshame
Source: StackOverflow