Installing Confluent Plugins on Kubernetes

1/25/2019

Our team is developing Kafka Connect Source Connector-plugins.

Do you have any ideas on how to install/upgrade the plugins? How is the flow (git -> Jenkins -> running Source Connector) supposed to look on-prem?

We use Confluent on Kubernetes which complicates things further.

PS. We are required by law to not use cloud solutions.

-- Daniel O
apache-kafka-connect
jenkins
jenkins--x
kubernetes

1 Answer

1/25/2019

To store custom connectors, use Nexus, Artifactory, S3, or some plain HTTP/file server.

If you are using Kubernetes, then you probably have a release policy around your Docker images.

Therefore, you can extend the Confluent Connect Docker images by adding additional RUN statements to the Dockerfile, then build and tag your images with Jenkins, and upgrade your Kubernetes services to use the new image tag.


The answer I would give for a bare-metal (or cloud) installation of managing Kafka Connect would be to use Ansible or other orchestration tool to push out the new files, and restart the services

-- OneCricketeer
Source: StackOverflow