Issue:

1. Shovel deployment failing as it's unable to communicate with kafka, getting the error: NoBrokersAvailable kafka=kafka.datacenter.svc.cluster.local:9092

ubuntu@k8s-worker-1:~$ sudo docker logs k8s_shovel-8469fc4b6d-ftmt4_03933125-ff8c-11e8-ac37-fa163e8dcff2_0
2018-12-14 10:35.48 Connected to graphite at grafana.datacenter.svc.cluster.local:2004
2018-12-14 10:35.48 connect-to-kafka kafka=kafka.datacenter.svc.cluster.local
2018-12-14 10:35.50 failed-to-connect-to-kafka e=NoBrokersAvailable() kafka=kafka.datacenter.svc.cluster.local:9092

2. After the above statement, the container exits.

Environment:

1. Kafka deployed in a Kubernetes environment using yml file.
2. Trying to deploy Shovel in Kubernetes environment using yml file.
3. Kafka connecting successfully to zookeeper.

4. Passing the following arguments while deploying shovel:

   args: - "shovel/main.py"
   - "--kafka=kafka.$(NAMESPACE).svc.cluster.local:9092"
   - "--consul=consul:4000"
   - "--host=grafana.$(NAMESPACE).svc.cluster.local"
   - "--topic=datacenter.test"

ubuntu@k8s-master-1:~$ kubectl describe pod shovel-8469fc4b6d-ftmt4 -n datacenter

State:      Waiting
  Reason:       CrashLoopBackOff
Last State: Terminated
  Reason:       Error
  Exit Code:    1

Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 

Events: Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned shovel-8469fc4b6d-ftmt4 to k8s-worker-1
Normal SuccessfulMountVolume 10m kubelet, k8s-worker-1 MountVolume.SetUp succeeded for volume "default-token-pk"
Normal Created 9m (x4 over 10m) kubelet, k8s-worker-1 Created container
Normal Started 9m (x4 over 10m) kubelet, k8s-worker-1 Started container
Normal Pulling 8m (x5 over 10m) kubelet, k8s-worker-1 pulling image "docker-registry:8000/datacenter-shovel"
Normal Pulled 8m (x5 over 10m) kubelet, k8s-worker-1 Successfully pulled image "docker-registry:8000/datacenter-shovel"
Warning BackOff 12s (x44 over 10m) kubelet, k8s-worker-1 Back-off restarting failed container

++ As per the packet capture between kafka and shovel PODS:

1. We are able to see packets being exchanged. (Internet Protocol Version 4, Src: 192.168.1.81, Dst: 192.168.1.94)

2. Shovel POD is able to see the packets coming from kafka POD.

3. Kafka POD is able to send the packets to the shovel pod.

Any assistance would be much appreciated.