I have three pods in a Kubernetes node.

Pod-1: Single node Kafka broker
Pod-2: Single node Zookeeper
Pod-3: Several Spring Boot Kafka enabled applications

Earlier on we were on Docker with similar setup. After we moved to Kubernetes setup all our applications in Pod-3 started giving this kind of error

[2020-01-23 06:37:05.267] :: [INFO ] :: o.a.k.c.c.i.AbstractCoordinator.markCoordinatorUnknown.729 :: [Consumer clientId=consumer-2, groupId=OperationResponseConsumer] Group coordinator 9.x.x.x:31865 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
[2020-01-23 06:37:05.386] :: [INFO ] :: o.a.k.c.c.i.AbstractCoordinator.onSuccess.677 :: [Consumer clientId=consumer-2, groupId=OperationResponseConsumer] Discovered group coordinator 9.x.x.x:31865 (id: 2147483646 rack: null)

This keeps looping on and on, none of the consumers ever latch on to topics. At the best consumer-group attachment is intermittent. What could this be?

Kafka: 2.2 Spring-Boot: 2.1.5

---- edit

On broker side I keep getting these logs

[2020-01-23 07:34:15,083] INFO [GroupCoordinator 1]: Member consumer-2-d2102198-df08-4009-9ba2-85350a2e9240 in group openapi-handler-consumer has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2020-01-23 07:34:15,083] INFO [GroupCoordinator 1]: Preparing to rebalance group openapi-handler-consumer in state PreparingRebalance with old generation 3 (__consumer_offsets-24) (reason: removing member consumer-2-d2102198-df08-4009-9ba2-85350a2e9240 on heartbeat expiration) (kafka.coordinator.group.GroupCoordinator)

------------- edit

Here are deployment descriptors we used for Kafka

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: 2020-01-23T09:04:18Z
  generation: 1
  labels:
    app: kafka
  name: xip-generator-kafka
  namespace: xip-dev-env
  resourceVersion: "53066450"
  selfLink: /apis/extensions/v1beta1/namespaces/xip-dev-env/deployments/xip-generator-kafka
  uid: 5664311b-3dbf-11ea-b0f2-005056ae16bc
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: kafka
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: kafka
    spec:
      containers:
      - env:
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: dip-generator-kafka-svc
        - name: KAFKA_ADVERTISED_PORT
          value: "9092"
        - name: KAFKA_BROKER_ID
          value: "1"
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: xip-generator-zookeeper-svc:2181
        - name: KAFKA_NUM_PARTITIONS
          value: "3"
        image: wurstmeister/kafka
        imagePullPolicy: IfNotPresent
        name: kafka
        ports:
        - containerPort: 9092
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2020-01-23T09:04:18Z
    lastUpdateTime: 2020-01-23T09:04:21Z
    message: ReplicaSet "xip-generator-kafka-76cf94fcdb" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: 2020-01-27T07:24:36Z
    lastUpdateTime: 2020-01-27T07:24:36Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Kafka service yml

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2020-01-23T09:04:18Z
  labels:
    app: kafka
  name: xip-generator-kafka-svc
  namespace: xip-dev-env
  resourceVersion: "52499070"
  selfLink: /api/v1/namespaces/xip-dev-env/services/xip-generator-kafka-svc
  uid: 56575969-3dbf-11ea-b0f2-005056ae16bc
spec:
  clusterIP: 10.x.x.x
  externalTrafficPolicy: Cluster
  ports:
  - name: kafka-port
    nodePort: 31865
    port: 9092
    protocol: TCP
    targetPort: 9092
  selector:
    app: kafka
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

Consumer App deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: 
  creationTimestamp: 2020-01-27T06:13:38Z
  generation: 1
  labels:
    app: add-devops-template
  name: xip-generator-add-devops-template
  namespace: xip-dev-env
  resourceVersion: "53059193"
  selfLink: /apis/extensions/v1beta1/namespaces/xip-dev-env/deployments/xip-generator-add-devops-template
  uid: 287d2c61-40cc-11ea-b0f2-005056ae16bc
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: add-devops-template
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: add-devops-template
    spec:
      containers:
      - env:
        - name: kafka.serverURL
          value: xip-generator-kafka-svc:9092
        - name: dmb.generator.mount.path
          value: /opt/code-engine/generated-code
        - name: spring.kafka.bootstrap-servers
          value: xip-generator-kafka-svc:9092
        - name: spring.data.mongodb.host
          value: mongourl
        - name: spring.data.mongodb.port
          value: "27018"
        - name: spring.data.mongodb.username
          value: root
        - name: spring.data.mongodb.password
          value: xxxxxxxxxxxxxx
        - name: spring.data.mongodb.database
          value: code_engine_dev
        - name: spring.data.mongodb.authentication-database
          value: admin
        - name: LOGSTASH
          value: logurl
        - name: vaultURL
          value: vaulturl
        - name: vaultSecret
          value: adminvault
        - name: vaultSecretFileName
          value: namefile
        - name: templateName
          value: add-devops.template
        - name: gitRepoToken
          value: token
        - name: gitRepoUserName
          value: username
        - name: templateDeploymentDetailsRepoBranch
          value: development
        - name: templateDeploymentDetailsRepoUrl
          value: url.git
        image: imagename
        imagePullPolicy: IfNotPresent
        name: add-devops-template
        ports:
        - containerPort: 8080
          protocol: TCP
        resources: {}
        securityContext:
          runAsUser: 0
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /opt/code-engine/generated-code
          name: xip-dev-pv-generatedcode
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: nexusegistrykey
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: xip-dev-pv-generatedcode
        persistentVolumeClaim:
          claimName: xip-dev-generatedcode-pvc
      - name: xip-dev-pv-logs
        persistentVolumeClaim:
          claimName: xip-dev-logs-pvc
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2020-01-27T06:13:57Z
    lastUpdateTime: 2020-01-27T06:13:57Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: 2020-01-27T06:13:38Z
    lastUpdateTime: 2020-01-27T06:13:57Z
    message: ReplicaSet "xip-generator-add-devops-template-7df595864c" has successfully
      progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1