How do we build a kafka image and then deploy it to kubernetes through the Kompose tool?
I was using this docker-compose.yml file. It works with docker-compose to build a kafka image. But when we use Kompose, which converts docker-compose.yml to kubernetes service, pvc, and deployments, not having the pre-built image is causing issues.
Building kafka image, then referencing it in the yml file also causes issues. Lastly, I am not able to mount volumes.
How can I fix these issues? Thank you
docker-compose.yml
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
build: .
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: myIP
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
links:
- zookeeper:zk
volumes:
- /var/run/docker.sock:/var/run/docker.sock
myApp:
image: myAppImage
ports:
- "3904:3904"
- "3905:3905"
volumes:
- /var/tmp/My.properties:/appl/myApp/bundleconfig/etc/appprops/My.properties
depends_on:
- zookeeper
- kafka
download-kafka.sh file
#!/bin/sh
mirror=$(curl --stderr /dev/null https://www.apache.org/dyn/closer.cgi\?as_json\=1 | jq -r '.preferred')
url="${mirror}kafka/${KAFKA_VERSION}/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz"
wget -q "${url}" -O "/tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz"
start-kafka.sh
#!/bin/bash
if [[ -z "$KAFKA_PORT" ]]; then
export KAFKA_PORT=9092
fi
if [[ -z "$KAFKA_ADVERTISED_PORT" ]]; then
export KAFKA_ADVERTISED_PORT=$(docker port `hostname` $KAFKA_PORT | sed -r "s/.*:(.*)/\1/g")
fi
if [[ -z "$KAFKA_BROKER_ID" ]]; then
# By default auto allocate broker ID
export KAFKA_BROKER_ID=1
fi
if [[ -z "$KAFKA_LOG_DIRS" ]]; then
export KAFKA_LOG_DIRS="/kafka/kafka-logs-$HOSTNAME"
fi
if [[ -z "$KAFKA_ZOOKEEPER_CONNECT" ]]; then
export KAFKA_ZOOKEEPER_CONNECT=$(env | grep ZK.*PORT_2181_TCP= | sed -e 's|.*tcp://||' | paste -sd ,)
fi
if [[ -n "$KAFKA_HEAP_OPTS" ]]; then
sed -r -i "s/(export KAFKA_HEAP_OPTS)=\"(.*)\"/\1=\"$KAFKA_HEAP_OPTS\"/g" $KAFKA_HOME/bin/kafka-server-start.sh
unset KAFKA_HEAP_OPTS
fi
if [[ -z "$KAFKA_ADVERTISED_HOST_NAME" && -n "$HOSTNAME_COMMAND" ]]; then
export KAFKA_ADVERTISED_HOST_NAME=$(eval $HOSTNAME_COMMAND)
fi
for VAR in `env`
do
if [[ $VAR =~ ^KAFKA_ && ! $VAR =~ ^KAFKA_HOME ]]; then
kafka_name=`echo "$VAR" | sed -r "s/KAFKA_(.*)=.*/\1/g" | tr '[:upper:]' '[:lower:]' | tr _ .`
env_var=`echo "$VAR" | sed -r "s/(.*)=.*/\1/g"`
if egrep -q "(^|^#)$kafka_name=" $KAFKA_HOME/config/server.properties; then
sed -r -i "s@(^|^#)($kafka_name)=(.*)@\2=${!env_var}@g" $KAFKA_HOME/config/server.properties #note that no config values may contain an '@' char
else
echo "$kafka_name=${!env_var}" >> $KAFKA_HOME/config/server.properties
fi
fi
done
if [[ -n "$CUSTOM_INIT_SCRIPT" ]] ; then
eval $CUSTOM_INIT_SCRIPT
fi
KAFKA_PID=0
# see https://medium.com/@gchudnov/trapping-signals-in-docker-containers-7a57fdda7d86#.bh35ir4u5
term_handler() {
echo 'Stopping Kafka....'
if [ $KAFKA_PID -ne 0 ]; then
kill -s TERM "$KAFKA_PID"
wait "$KAFKA_PID"
fi
echo 'Kafka stopped.'
exit
}
# Capture kill requests to stop properly
trap "term_handler" SIGHUP SIGINT SIGTERM
create-topics.sh &
$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &
KAFKA_PID=$!
Step 1: Do docker-compose build
and you will have a kafka
image that you can use to deploy on the Kubernetes. But to enable Kubernetes to pull this image either push it docker hub or some private docker registry from where your Kubernetes cluster can pull images.
Note the name of the docker image from the above step since that is needed in step below.
Also if you push the image to docker hub it will be named as docker.io/yourUserName/imageName
but if you push to some other docker registry it will be named differently.
Step 2: You have to change the docker-compose file in the kafka
service to include the image name you gave above. This is an issue in kompose where in it should error out since the image name is not given and kompose cannot magically give some image name in the generated Kubernetes config.
So change it to something like this
kafka:
build: .
image: docker.io/username/mykafka:0.1
ports:
- "9092:9092"
Step 3: Once you add that image name you will have a valid Kubernetes config generated by kompose with the image name you have provided.
Now you can try to deploy the application and Kubernetes will be able to pull the image just fine.
If you are doing this in a single node cluster like minikube there is a easy workaround you can follow as mentioned in this blog.
configMaps