I have docker container (Hadoop installation https://github.com/kiwenlau/hadoop-cluster-docker) that I can run using sudo docker run -itd -p 50070:50070 -p 8088:8088 --name hadoop-master kiwenlau/hadoop:1.0 command without any issue, however when trying to deploy same image to kubernetes, pod failing to start. In order to create deployment I'm using kubectl run hadoop-master --image=kiwenlau/hadoop:1.0 --port=8088 --port=50070 command
Here log of describe pod command
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
6m 6m 1 default-scheduler Normal Scheduled Successfully assigned hadoop-master-2828539450-rnwsd to gke-mtd-cluster-default-pool-6b97d4d0-hcbt
6m 6m 1 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Normal Created Created container with id 1560ff87e0e7357c76cec89f5f429e0b9b5fc51523c79e4e2c12df1834d7dd75
6m 6m 1 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Normal Started Started container with id 1560ff87e0e7357c76cec89f5f429e0b9b5fc51523c79e4e2c12df1834d7dd75
6m 6m 1 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Normal Created Created container with id c939d3336687a33e69d37aa73177e673fd56d766cb499a4235e89d554d233c37
6m 6m 1 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Normal Started Started container with id c939d3336687a33e69d37aa73177e673fd56d766cb499a4235e89d554d233c37
6m 6m 2 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "hadoop-master" with CrashLoopBackOff: "Back-off 10s restarting failed container=hadoop-master pod=hadoop-master-2828539450-rnwsd_default(562dae39-a757-11e7-a5a3-42010a8401c6)"
6m 6m 1 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Normal Created Created container with id 7d1c67686c039e459ee0ea3936eedb4996a5201f6a1fec02ac98d219bb07745f
6m 6m 1 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Normal Started Started container with id 7d1c67686c039e459ee0ea3936eedb4996a5201f6a1fec02ac98d219bb07745f
6m 6m 2 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "hadoop-master" with CrashLoopBackOff: "Back-off 20s restarting failed container=hadoop-master pod=hadoop-master-2828539450-rnwsd_default(562dae39-a757-11e7-a5a3-42010a8401c6)"
5m 5m 1 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Normal Started Started container with id a8879a2c794b3e62f788ad56e403cb619644e9219b2c092e760ddeba506b2e44
5m 5m 1 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Normal Created Created container with id a8879a2c794b3e62f788ad56e403cb619644e9219b2c092e760ddeba506b2e44
5m 5m 3 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "hadoop-master" with CrashLoopBackOff: "Back-off 40s restarting failed container=hadoop-master pod=hadoop-master-2828539450-rnwsd_default(562dae39-a757-11e7-a5a3-42010a8401c6)"
5m 5m 1 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Normal Created Created container with id 8907cdf19c51b87cea6e1e611649e874db2c21f47234df54bd9f27515cee0a0e
5m 5m 1 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Normal Started Started container with id 8907cdf19c51b87cea6e1e611649e874db2c21f47234df54bd9f27515cee0a0e
5m 3m 7 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "hadoop-master" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=hadoop-master pod=hadoop-master-2828539450-rnwsd_default(562dae39-a757-11e7-a5a3-42010a8401c6)"
3m 3m 1 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Normal Created Created container with id 294072caea596b47324914a235c1882dbc521cc355644a1e25ebf06f0e04301f
3m 3m 1 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Normal Started Started container with id 294072caea596b47324914a235c1882dbc521cc355644a1e25ebf06f0e04301f
3m 1m 12 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "hadoop-master" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=hadoop-master pod=hadoop-master-2828539450-rnwsd_default(562dae39-a757-11e7-a5a3-42010a8401c6)"
6m 50s 7 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Normal Pulled Container image "kiwenlau/hadoop:1.0" already present on machine
50s 50s 1 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Normal Created Created container with id 7da7508ac864d04d47639b0d2c374a27c3e8a3351e13a2564e57453cf857426d
50s 50s 1 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Normal Started Started container with id 7da7508ac864d04d47639b0d2c374a27c3e8a3351e13a2564e57453cf857426d
6m 0s 31 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt spec.containers{hadoop-master} Warning BackOff Back-off restarting failed container
49s 0s 5 kubelet, gke-mtd-cluster-default-pool-6b97d4d0-hcbt Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "hadoop-master" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=hadoop-master pod=hadoop-master-2828539450-rnwsd_default(562dae39-a757-11e7-a5a3-42010a8401c6)"kubectl log output:
kubectl logs hadoop-master-2828539450-rnwsd
* Starting OpenBSD Secure Shell server sshd
...done.Note that docker container itself not starting hadoop. In order to start it, I have to connect to container and start hadoop manually, however I want to be able to simply run container in K8s for now.
Thanks
The equivalent command in kubernetes is
kubectl run -it hadoop-master --image=kiwenlau/hadoop:1.0 --port=8088 --port=50070Actually it's not about kubernetes, there is something wrong in the Dockerfile.
A docker container exits when its main process finishes.
With CMD [ "sh", "-c", "service ssh start; bash"], SSH service is started at the background, after then the container stopped as job finished.
It should be an executable script/program running in the foreground always, like ~/start-hadoop.sh.
FYI, usually we don't need ssh to the container, as docker exec -it some_container bash should be enough.