strimzi 0.14: Kafka broker failed authentication

10/31/2019

I tried to build a Kafka cluster using Strimzi (0.14) in a Kubernetes cluster.

I use the examples come with the strimzi, i.e. examples/kafka/kafka-persistent.yaml.

This yaml file looks like this:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 2.3.0
    replicas: 3
    listeners:
      plain: {}
      tls: {}
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.3"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 12Gi
        deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 9Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

kubectl apply -f examples/kafka/kafka-persistent.yaml

Both zookeepers and kafka brokers were brought up.
However, I saw errors in kafka broker logs: [SocketServer brokerId=1] Failed authentication with /10.244.5.94 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-1-ListenerName(REPLICATION)-SSL-0]

Anyone know how to fix the problem?

-- Kevin Z
apache-kafka
kubernetes
strimzi

1 Answer

11/8/2019

One of the things which can cause this is if your cluster is using different DNS suffix for service domains (default is .cluster.local). You need to find out the right DNS suffix and use the environment variable KUBERNETES_SERVICE_DNS_DOMAIN in the Strimzi Cluster Operator deployment to override the default value.

If you exec into one of the Kafka or Zookeeper pods and do hostname -f it should show you the full hostname from which you can identifythe suffix.

(I pasted this from comments as a full answer since it helped to solve the question.)

-- Jakub
Source: StackOverflow