I would like to run the Kafka connect workers on OpenShift. I would like to use OpenShift to (auto-)scale out. However, it is not clear to me how to pass the connector configuration if it is usually provided manually via a REST call to a particular worker after it has started.
Thus, the question is if it is possible to provide a complete specification of a connector to the worker at the start up time without the REST calls.
at the start up time without the REST calls
No, it is not. See the issue I created for it.
You have to load the conncetor configuration only once with REST, though. The worker configuration is entirely environment variable driven, and scaling workers doesn't require posting new connector configuration to scale them unless you did want to increase the tasks.max for example. But you can just start this as the number of Kafka partitions anyway, and that is your limit for Connect, even with a single worker.
Personally, I think you can inject the information to the kafka pods using some templates, such as deploymentConfig, statefulset, daemonset and so on.
For example, the following link is explained about a ruby application how to get the mysql connection information from injecting environment variables.
Defining a Service for an External Database
env:
-
name: "MYSQL_USER"
value: "${MYSQL_USER}"
-
name: "MYSQL_PASSWORD"
value: "${MYSQL_PASSWORD}"
-
name: "MYSQL_DATABASE"
value: "${MYSQL_DATABASE}" Like above example, you should start some wrapper start script or binary of related application which are configured by getting the connection information as environment variables.