Cadence cluster not coming up on Kubernetes

11/20/2019

I am trying get cadence running on a kubernetes cluster. However, I noticed there's a bug in the Cadence server initialization which prevents Cassandra scripts from running to properly initialize schema. https://github.com/uber/cadence/issues/1713 So I thought I would do this step manually. I performed the following steps -

  • kompose convert on the docker compose from https://raw.githubusercontent.com/uber/cadence/master/docker/docker-compose.yml
  • kubectl apply -f cassandra-deployment.yaml
  • Run the container locally copy the cadence-cassandra-tool(usr/local/bin/cadence-cassandra-tool) and cassandra(/etc/cadence/schema/cassandra) folder from local container to local directory
  • Copy these two files to cassandra pods
  • login into the cassandra pod
  • Run the following command on the container
/cadence-cassandra-tool --ep 127.0.0.1 -k cadence setup-schema -v 0.0
/cadence-cassandra-tool --ep 127.0.0.1 -k cadence update-schema -d /cassandra/cadence/versioned
/cadence-cassandra-tool --ep 127.0.0.1 create -k cadence_visibility --rf 1
/cadence-cassandra-tool --ep 127.0.0.1 -k cadence_visibility setup-schema -v 0.0
/cadence-cassandra-tool --ep 127.0.0.1 -k cadence_visibility update-schema -d /cassandra/visibility/versioned
exit

At this point I checked describe keyspaces; via. cqlsh and I don't see these keyspaces created in cassandra!

  • Run cassandra service, stastd, cadence-service, cadence-web and expose it to cluster. What i noticed is that cadence service is up and I can get on the UI. We have exposed the cadence-service and cadence-web using Load balancer so I can reach the IP from my local.

What I noticed is the cadence-service pod is not running. I get the following error -

cassandra started
2019/11/19 23:39:30 Loading config; env=docker,zone=,configDir=/etc/cadence/config
2019/11/19 23:39:30 Loading configFiles=[/etc/cadence/config/docker.yaml]
2019/11/19 23:39:30 error: failed to connect to 10.196.243.88:9042 due to error: Keyspace 'cadence' does not exist
2019/11/19 23:39:30 error: failed to connect to 10.196.13.25:9042 due to error: Keyspace 'cadence' does not exist
2019/11/19 23:39:30 error: failed to connect to 10.196.243.88:9042 due to error: Keyspace 'cadence' does not exist
2019/11/19 23:39:30 Incompatible versionsunable to create CQL Client: no connections were made when creating the session

Any pointers on how to get it up? Thanks.

-- cheddarDev
cadence-workflow
kubernetes

1 Answer

12/10/2019

There is a helm repo https://hub.helm.sh/charts/banzaicloud-stable/cadence which helped solved this issue.

-- cheddarDev
Source: StackOverflow