I need to run Kafka in a local Kubernetes instance (using Minikube) and to have the resulting Kafka service accessible to client applications (publishers and subscribers) outside the Minikube VM.
I have everything up and running in Minikube but I suppose that I have made a configuration mistake since I cannot access Kafka from outside. I have read similar questions and tried there suggested solutions but none of them solved the issue for me.
I have posted my YAML configuration files at https://github.com/thomasleplus/docker-kafka as well as the shell script that I am using to start the whole thing on my Ubuntu machine. I would really appreciate it if someone could help me spot what I have missed.
Here's my configuration so far:
$ kubectl describe service kafka-service
Name: kafka-service
Namespace: default
Labels: run=kafka
Annotations: <none>
Selector: run=kafka
Type: NodePort
IP: 10.0.0.121
Port: kafka-port 30123/TCP
NodePort: kafka-port 30123/TCP
Endpoints: 172.17.0.3:9092
Session Affinity: None
Events: <none>
$ kubectl describe deployment kafka-deployment
Name: kafka-deployment
Namespace: default
CreationTimestamp: Thu, 17 Aug 2017 20:42:51 -0700
Labels: run=kafka
Annotations: deployment.kubernetes.io/revision=1
Selector: run=kafka
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 1 max surge
Pod Template:
Labels: run=kafka
Containers:
kafka-service:
Image: wurstmeister/kafka
Port: 9092/TCP
Environment:
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
KAFKA_ADVERTISED_PORT: 30123
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper-service:2181
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
OldReplicaSets: <none>
NewReplicaSet: kafka-deployment-2817439001 (1/1 replicas created)
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
15m 15m 1 deployment-controller Normal ScalingReplicaSet Scaled up replica set kafka-deployment-2817439001 to 1The logs:
waiting for kafka to be ready
[2017-08-18 04:31:00,296] INFO KafkaConfig values:
advertised.host.name = null
advertised.listeners = PLAINTEXT://192.168.99.100:30123
advertised.port = null
alter.config.policy.class.name = null
authorizer.class.name =
auto.create.topics.enable = true
auto.leader.rebalance.enable = true
background.threads = 10
broker.id = 1
broker.id.generation.enable = true
broker.rack = null
compression.type = producer
connections.max.idle.ms = 600000
controlled.shutdown.enable = true
controlled.shutdown.max.retries = 3
controlled.shutdown.retry.backoff.ms = 5000
controller.socket.timeout.ms = 30000
create.topic.policy.class.name = null
default.replication.factor = 1
delete.records.purgatory.purge.interval.requests = 1
delete.topic.enable = false
fetch.purgatory.purge.interval.requests = 1000
group.initial.rebalance.delay.ms = 0
group.max.session.timeout.ms = 300000
group.min.session.timeout.ms = 6000
host.name =
inter.broker.listener.name = null
inter.broker.protocol.version = 0.11.0-IV2
leader.imbalance.check.interval.seconds = 300
leader.imbalance.per.broker.percentage = 10
listener.security.protocol.map = SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,TRACE:TRACE,SASL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT
listeners = PLAINTEXT://:9092
log.cleaner.backoff.ms = 15000
log.cleaner.dedupe.buffer.size = 134217728
log.cleaner.delete.retention.ms = 86400000
log.cleaner.enable = true
log.cleaner.io.buffer.load.factor = 0.9
log.cleaner.io.buffer.size = 524288
log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
log.cleaner.min.cleanable.ratio = 0.5
log.cleaner.min.compaction.lag.ms = 0
log.cleaner.threads = 1
log.cleanup.policy = [delete]
log.dir = /tmp/kafka-logs
log.dirs = /kafka/kafka-logs-kafka-deployment-2817439001-tqbjq
log.flush.interval.messages = 9223372036854775807
log.flush.interval.ms = null
log.flush.offset.checkpoint.interval.ms = 60000
log.flush.scheduler.interval.ms = 9223372036854775807
log.flush.start.offset.checkpoint.interval.ms = 60000
log.index.interval.bytes = 4096
log.index.size.max.bytes = 10485760
log.message.format.version = 0.11.0-IV2
log.message.timestamp.difference.max.ms = 9223372036854775807
log.message.timestamp.type = CreateTime
log.preallocate = false
log.retention.bytes = -1
log.retention.check.interval.ms = 300000
log.retention.hours = 168
log.retention.minutes = null
log.retention.ms = null
log.roll.hours = 168
log.roll.jitter.hours = 0
log.roll.jitter.ms = null
log.roll.ms = null
log.segment.bytes = 1073741824
log.segment.delete.delay.ms = 60000
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides =
message.max.bytes = 1000012
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
min.insync.replicas = 1
num.io.threads = 8
num.network.threads = 3
num.partitions = 1
num.recovery.threads.per.data.dir = 1
num.replica.fetchers = 1
offset.metadata.max.bytes = 4096
offsets.commit.required.acks = -1
offsets.commit.timeout.ms = 5000
offsets.load.buffer.size = 5242880
offsets.retention.check.interval.ms = 600000
offsets.retention.minutes = 1440
offsets.topic.compression.codec = 0
offsets.topic.num.partitions = 50
offsets.topic.replication.factor = 1
offsets.topic.segment.bytes = 104857600
port = 9092
principal.builder.class = class org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
producer.purgatory.purge.interval.requests = 1000
queued.max.requests = 500
quota.consumer.default = 9223372036854775807
quota.producer.default = 9223372036854775807
quota.window.num = 11
quota.window.size.seconds = 1
replica.fetch.backoff.ms = 1000
replica.fetch.max.bytes = 1048576
replica.fetch.min.bytes = 1
replica.fetch.response.max.bytes = 10485760
replica.fetch.wait.max.ms = 500
replica.high.watermark.checkpoint.interval.ms = 5000
replica.lag.time.max.ms = 10000
replica.socket.receive.buffer.bytes = 65536
replica.socket.timeout.ms = 30000
replication.quota.window.num = 11
replication.quota.window.size.seconds = 1
request.timeout.ms = 30000
reserved.broker.max.id = 1000
sasl.enabled.mechanisms = [GSSAPI]
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.principal.to.local.rules = [DEFAULT]
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism.inter.broker.protocol = GSSAPI
security.inter.broker.protocol = PLAINTEXT
socket.receive.buffer.bytes = 102400
socket.request.max.bytes = 104857600
socket.send.buffer.bytes = 102400
ssl.cipher.suites = null
ssl.client.auth = none
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.abort.timed.out.transaction.cleanup.interval.ms = 60000
transaction.max.timeout.ms = 900000
transaction.remove.expired.transaction.cleanup.interval.ms = 3600000
transaction.state.log.load.buffer.size = 5242880
transaction.state.log.min.isr = 1
transaction.state.log.num.partitions = 50
transaction.state.log.replication.factor = 1
transaction.state.log.segment.bytes = 104857600
transactional.id.expiration.ms = 604800000
unclean.leader.election.enable = false
zookeeper.connect = zookeeper-service:2181
zookeeper.connection.timeout.ms = 6000
zookeeper.session.timeout.ms = 6000
zookeeper.set.acl = false
zookeeper.sync.time.ms = 2000
(kafka.server.KafkaConfig)
[2017-08-18 04:31:00,436] INFO starting (kafka.server.KafkaServer)
[2017-08-18 04:31:00,439] INFO Connecting to zookeeper on zookeeper-service:2181 (kafka.server.KafkaServer)
[2017-08-18 04:31:00,467] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2017-08-18 04:31:00,472] INFO Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,472] INFO Client environment:host.name=kafka-deployment-2817439001-tqbjq (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,473] INFO Client environment:java.version=1.8.0_131 (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,473] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,473] INFO Client environment:java.home=/opt/jdk1.8.0_131/jre (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,473] INFO Client environment:java.class.path=:/opt/kafka/bin/../libs/aopalliance-repackaged-2.5.0-b05.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/commons-lang3-3.5.jar:/opt/kafka/bin/../libs/connect-api-0.11.0.0.jar:/opt/kafka/bin/../libs/connect-file-0.11.0.0.jar:/opt/kafka/bin/../libs/connect-json-0.11.0.0.jar:/opt/kafka/bin/../libs/connect-runtime-0.11.0.0.jar:/opt/kafka/bin/../libs/connect-transforms-0.11.0.0.jar:/opt/kafka/bin/../libs/guava-20.0.jar:/opt/kafka/bin/../libs/hk2-api-2.5.0-b05.jar:/opt/kafka/bin/../libs/hk2-locator-2.5.0-b05.jar:/opt/kafka/bin/../libs/hk2-utils-2.5.0-b05.jar:/opt/kafka/bin/../libs/jackson-annotations-2.8.5.jar:/opt/kafka/bin/../libs/jackson-core-2.8.5.jar:/opt/kafka/bin/../libs/jackson-databind-2.8.5.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.8.5.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.8.5.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.8.5.jar:/opt/kafka/bin/../libs/javassist-3.21.0-GA.jar:/opt/kafka/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/bin/../libs/javax.inject-1.jar:/opt/kafka/bin/../libs/javax.inject-2.5.0-b05.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.0.1.jar:/opt/kafka/bin/../libs/jersey-client-2.24.jar:/opt/kafka/bin/../libs/jersey-common-2.24.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.24.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.24.jar:/opt/kafka/bin/../libs/jersey-guava-2.24.jar:/opt/kafka/bin/../libs/jersey-media-jaxb-2.24.jar:/opt/kafka/bin/../libs/jersey-server-2.24.jar:/opt/kafka/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jetty-http-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jetty-io-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jetty-security-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jetty-server-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jetty-util-9.2.15.v20160210.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.3.jar:/opt/kafka/bin/../libs/kafka-clients-0.11.0.0.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-0.11.0.0.jar:/opt/kafka/bin/../libs/kafka-streams-0.11.0.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-0.11.0.0.jar:/opt/kafka/bin/../libs/kafka-tools-0.11.0.0.jar:/opt/kafka/bin/../libs/kafka_2.12-0.11.0.0-sources.jar:/opt/kafka/bin/../libs/kafka_2.12-0.11.0.0-test-sources.jar:/opt/kafka/bin/../libs/kafka_2.12-0.11.0.0.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-1.3.0.jar:/opt/kafka/bin/../libs/maven-artifact-3.5.0.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/bin/../libs/plexus-utils-3.0.24.jar:/opt/kafka/bin/../libs/reflections-0.9.11.jar:/opt/kafka/bin/../libs/rocksdbjni-5.0.1.jar:/opt/kafka/bin/../libs/scala-library-2.12.2.jar:/opt/kafka/bin/../libs/scala-parser-combinators_2.12-1.0.4.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.25.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/bin/../libs/snappy-java-1.1.2.6.jar:/opt/kafka/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/bin/../libs/zkclient-0.10.jar:/opt/kafka/bin/../libs/zookeeper-3.4.10.jar (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,473] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,473] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,473] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,473] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,473] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,474] INFO Client environment:os.version=4.9.13 (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,474] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,474] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,474] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,475] INFO Initiating client connection, connectString=zookeeper-service:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@6d2a209c (org.apache.zookeeper.ZooKeeper)
[2017-08-18 04:31:00,500] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2017-08-18 04:31:00,505] INFO Opening socket connection to server zookeeper-service.default.svc.cluster.local/10.0.0.56:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2017-08-18 04:31:00,516] INFO Socket connection established to zookeeper-service.default.svc.cluster.local/10.0.0.56:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2017-08-18 04:31:00,558] INFO Session establishment complete on server zookeeper-service.default.svc.cluster.local/10.0.0.56:2181, sessionid = 0x15df39b70410000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2017-08-18 04:31:00,560] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2017-08-18 04:31:00,682] INFO Cluster ID = V2Mj7cI3SMG_VoQxmtb9Tw (kafka.server.KafkaServer)
[2017-08-18 04:31:00,685] WARN No meta.properties file under dir /kafka/kafka-logs-kafka-deployment-2817439001-tqbjq/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2017-08-18 04:31:00,727] INFO [ThrottledRequestReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-08-18 04:31:00,727] INFO [ThrottledRequestReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-08-18 04:31:00,728] INFO [ThrottledRequestReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-08-18 04:31:00,801] INFO Log directory '/kafka/kafka-logs-kafka-deployment-2817439001-tqbjq' not found, creating it. (kafka.log.LogManager)
[2017-08-18 04:31:00,810] INFO Loading logs. (kafka.log.LogManager)
[2017-08-18 04:31:00,819] INFO Logs loading complete in 9 ms. (kafka.log.LogManager)
[2017-08-18 04:31:00,891] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2017-08-18 04:31:00,899] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2017-08-18 04:31:00,960] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2017-08-18 04:31:00,965] INFO [Socket Server on Broker 1], Started 1 acceptor threads (kafka.network.SocketServer)
[2017-08-18 04:31:00,982] INFO [ExpirationReaper-1-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-18 04:31:00,985] INFO [ExpirationReaper-1-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-18 04:31:00,989] INFO [ExpirationReaper-1-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-18 04:31:01,089] INFO [ExpirationReaper-1-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-18 04:31:01,090] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2017-08-18 04:31:01,101] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2017-08-18 04:31:01,101] INFO [ExpirationReaper-1-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-18 04:31:01,128] INFO [ExpirationReaper-1-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-08-18 04:31:01,164] INFO [GroupCoordinator 1]: Starting up. (kafka.coordinator.group.GroupCoordinator)
[2017-08-18 04:31:01,170] INFO [GroupCoordinator 1]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
[2017-08-18 04:31:01,178] INFO [Group Metadata Manager on Broker 1]: Removed 0 expired offsets in 11 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2017-08-18 04:31:01,200] INFO [ProducerId Manager 1]: Acquired new producerId block (brokerId:1,blockStartProducerId:0,blockEndProducerId:999) by writing to Zk with path version 1 (kafka.coordinator.transaction.ProducerIdManager)
[2017-08-18 04:31:01,259] INFO [Transaction Coordinator 1]: Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2017-08-18 04:31:01,277] INFO [Transaction Coordinator 1]: Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2017-08-18 04:31:01,277] INFO [Transaction Marker Channel Manager 1]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2017-08-18 04:31:01,335] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2017-08-18 04:31:01,374] INFO Creating /brokers/ids/1 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2017-08-18 04:31:01,378] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2017-08-18 04:31:01,381] INFO Registered broker 1 at path /brokers/ids/1 with addresses: EndPoint(192.168.99.100,30123,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils)
[2017-08-18 04:31:01,383] WARN No meta.properties file under dir /kafka/kafka-logs-kafka-deployment-2817439001-tqbjq/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2017-08-18 04:31:01,394] INFO Kafka version : 0.11.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2017-08-18 04:31:01,394] INFO Kafka commitId : cb8625948210849f (org.apache.kafka.common.utils.AppInfoParser)
[2017-08-18 04:31:01,395] INFO [Kafka Server 1], started (kafka.server.KafkaServer)
[2017-08-18 04:41:01,167] INFO [Group Metadata Manager on Broker 1]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)Most answers to similar questions recommend to use the service type NodePort as I do. And to use port/targetPort/nodePort to map the default 9092 port of Kafka to an exposable port (I chose 30123).
$ minikube service kafka-service --url
http://192.168.99.100:30123
$ nmap 192.168.99.100 -p 30123
Starting Nmap 7.40 ( https://nmap.org ) at 2017-08-17 20:43 PDT
Nmap scan report for 192.168.99.100
Host is up (0.00036s latency).
PORT STATE SERVICE
30123/tcp open unknown
Nmap done: 1 IP address (1 host up) scanned in 13.06 secondsIn the end, it looks like 192.168.99.100:30123 should be the way to access Kafka from outside Minikube (so that's what I've put in KAFKA_ADVERTISED_HOST_NAME and KAFKA_ADVERTISED_PORT) yet clients can't connect using these:
$ kafkacat -C -b 192.168.99.100:30123 -t demo
% ERROR: Topic demo error: Broker: Leader not availableFinally some answers mention potential firewall interference so I have tried disabling my machine's firewall but it didn't change anything. If I need to disable the firewall inside the Minikube VM, I am not sure how to do that.
Any help would be greatly appreciated.
The problem is to do with a bug in recent versions of minikube (see https://github.com/kubernetes/minikube/issues/1690).
The solution is simply:
minikube ssh sudo ip link set docker0 promisc on
I know that this is an old post but I have had a similar "Leader not available" issue that I faced with almost a similar kafka + zookeeper deployment and service within minikube. The crux of this issue is that zookeeper is not able to talk to kafka within minikube.
The easiest fix for you here would be to revert to an earlier compatible version of minikube: v0.17.1 (that uses kubernetes version v1.5.3). If this is not an option for you, the possible workarounds that worked for me in minikube versions v0.19.0, v0.21.0 and v0.23.0 were the following for me:
minikube start --kubernetes-version v1.5.3Secondly, the issue was that I was using the KAFKA_ZOOKEEPER_CONNECT: zookeeper-service:2181 like you. Whilst this setting seemed to work fine if minikube was started with kubernetes client and server version v1.5.3 (i.e. using minikube version v0.17.1), this setting seemed to fail for kubernetes server version v1.7.0 (i.e. using minikube versions v0.21.0 and v0.23.0). The workaround was to expose the zookeeper service using type NodePort like so:
apiVersion: v1
kind: Service
metadata:
labels:
app: zookeeper-service
name: zookeeper-service
spec:
type: NodePort
ports:
- name: zookeeper-port
port: 2181
nodePort: 30181
targetPort: 2181
selector:
app: zookeeperand updating the kafka-deployment.yml like so:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: kafka
name: kafka
spec:
replicas: 1
template:
metadata:
labels:
app: kafka
spec:
containers:
- env:
- name: KAFKA_ADVERTISED_HOST_NAME
value: "192.168.99.100"
- name: KAFKA_ADVERTISED_PORT
value: "30123"
- name: KAFKA_BROKER_ID
value: "1"
- name: KAFKA_ZOOKEEPER_CONNECT
value: 192.168.99.100:30181
- name: KAFKA_CREATE_TOPICS
value: "demo:1:1"
image: wurstmeister/kafka
imagePullPolicy: Always
name: kafka
ports:
- containerPort: 9092echo "Am I receiving this message?" | kafkacat -P -b 192.168.99.100:30123 -t demo
kafkacat -C -b 192.168.99.100:30123 -t demo
% Reached end of topic demo [0] at offset 0
Am I receiving this message?
Unfortunately, the latest minikube version v0.25.0 (that uses kubernetes version v1.9.0) does not support downgrading the kubernetes version and this workaround does not work if you are using the latest minikube version.
If anyone else finds a better solution on this issue, please do update this thread!
Thanks.