create pods staying at ContainerCreating

12/25/2018

Check kubelet, no error message is found, specific kubelet log:

-- Logs begin at Fri 2018-12-21 20:08:48 CST. --
Dec 25 10:36:35 node-01 kubelet[1579]: I1225 10:36:35.414112    1579 factory.go:113] Factory "systemd" can handle container "/system.slice/-.mount", but ignoring.
Dec 25 10:36:35 node-01 kubelet[1579]: I1225 10:36:35.414117    1579 manager.go:971] ignoring container "/system.slice/-.mount"
Dec 25 10:36:35 node-01 kubelet[1579]: I1225 10:36:35.414120    1579 factory.go:120] Factory "docker" was unable to handle container "/system.slice/dev-hugepages.mount"
Dec 25 10:36:35 node-01 kubelet[1579]: I1225 10:36:35.414124    1579 factory.go:113] Factory "systemd" can handle container "/system.slice/dev-hugepages.mount", but ignoring.
Dec 25 10:36:35 node-01 kubelet[1579]: I1225 10:36:35.414129    1579 manager.go:971] ignoring container "/system.slice/dev-hugepages.mount"
Dec 25 10:36:35 node-01 kubelet[1579]: I1225 10:36:35.470988    1579 qos_container_manager_linux.go:338] [ContainerManager]: Updated QoS cgroup configuration
Dec 25 10:36:35 node-01 kubelet[1579]: I1225 10:36:35.995906    1579 kubelet.go:1290] Container garbage collection succeeded
Dec 25 10:36:36 node-01 kubelet[1579]: I1225 10:36:36.610912    1579 kubelet.go:2189] Container runtime status: Runtime Conditions: RuntimeReady=true reason: message:, NetworkReady=true reason: message:
Dec 25 10:36:37 node-01 kubelet[1579]: I1225 10:36:37.195887    1579 kubelet.go:1995] SyncLoop (housekeeping)
Dec 25 10:36:39 node-01 kubelet[1579]: I1225 10:36:39.195893    1579 kubelet.go:1995] SyncLoop (housekeeping)
Dec 25 10:36:40 node-01 kubelet[1579]: I1225 10:36:40.306442    1579 eviction_manager.go:226] eviction manager: synchronize housekeeping
Dec 25 10:36:40 node-01 kubelet[1579]: I1225 10:36:40.315282    1579 helpers.go:836] eviction manager: observations: signal=nodefs.inodesFree, available: 2576153, capacity: 2621440, time: 2018-12-25 10:36:40.307111589 +0800 CST m=+5045.843118543
Dec 25 10:36:40 node-01 kubelet[1579]: I1225 10:36:40.315318    1579 helpers.go:836] eviction manager: observations: signal=imagefs.available, available: 35152656Ki, capacity: 41151808Ki, time: 2018-12-25 10:36:40.307111589 +0800 CST m=+5045.843118543
Dec 25 10:36:40 node-01 kubelet[1579]: I1225 10:36:40.315329    1579 helpers.go:836] eviction manager: observations: signal=imagefs.inodesFree, available: 2576153, capacity: 2621440, time: 2018-12-25 10:36:40.307111589 +0800 CST m=+5045.843118543
Dec 25 10:36:40 node-01 kubelet[1579]: I1225 10:36:40.315335    1579 helpers.go:836] eviction manager: observations: signal=pid.available, available: 32597, capacity: 32Ki, time: 2018-12-25 10:36:40.314846776 +0800 CST m=+5045.850853716
Dec 25 10:36:40 node-01 kubelet[1579]: I1225 10:36:40.315352    1579 helpers.go:836] eviction manager: observations: signal=memory.available, available: 502688Ki, capacity: 1016168Ki, time: 2018-12-25 10:36:40.307111589 +0800 CST m=+5045.843118543
Dec 25 10:36:40 node-01 kubelet[1579]: I1225 10:36:40.315360    1579 helpers.go:836] eviction manager: observations: signal=allocatableMemory.available, available: 1014508Ki, capacity: 1016168Ki, time: 2018-12-25 10:36:40.315163272 +0800 CST m=+5045.851170184
Dec 25 10:36:40 node-01 kubelet[1579]: I1225 10:36:40.315366    1579 helpers.go:836] eviction manager: observations: signal=nodefs.available, available: 35152656Ki, capacity: 41151808Ki, time: 2018-12-25 10:36:40.307111589 +0800 CST m=+5045.843118543
Dec 25 10:36:40 node-01 kubelet[1579]: I1225 10:36:40.315391    1579 eviction_manager.go:317] eviction manager: no resources are starved
Dec 25 10:36:40 node-01 kubelet[1579]: I1225 10:36:40.319176    1579 setters.go:736] Error getting volume limit for plugin kubernetes.io/azure-disk
Dec 25 10:36:40 node-01 kubelet[1579]: I1225 10:36:40.319188    1579 setters.go:736] Error getting volume limit for plugin kubernetes.io/gce-pd
Dec 25 10:36:40 node-01 kubelet[1579]: I1225 10:36:40.319192    1579 setters.go:736] Error getting volume limit for plugin kubernetes.io/aws-ebs
Dec 25 10:36:41 node-01 kubelet[1579]: I1225 10:36:41.195905    1579 kubelet.go:1995] SyncLoop (housekeeping)
Dec 25 10:36:41 node-01 kubelet[1579]: I1225 10:36:41.612060    1579 kubelet.go:2189] Container runtime status: Runtime Conditions: RuntimeReady=true reason: message:, NetworkReady=true reason: message:
Dec 25 10:36:43 node-01 kubelet[1579]: I1225 10:36:43.195873    1579 kubelet.go:1995] SyncLoop (housekeeping)
Dec 25 10:36:45 node-01 kubelet[1579]: I1225 10:36:45.195888    1579 kubelet.go:1995] SyncLoop (housekeeping)
Dec 25 10:36:46 node-01 kubelet[1579]: I1225 10:36:46.613281    1579 kubelet.go:2189] Container runtime status: Runtime Conditions: RuntimeReady=true reason: message:, NetworkReady=true reason: message:
Dec 25 10:36:47 node-01 kubelet[1579]: I1225 10:36:47.195895    1579 kubelet.go:1995] SyncLoop (housekeeping)
Dec 25 10:36:49 node-01 kubelet[1579]: I1225 10:36:49.195876    1579 kubelet.go:1995] SyncLoop (housekeeping)
Dec 25 10:36:50 node-01 kubelet[1579]: I1225 10:36:50.315497    1579 eviction_manager.go:226] eviction manager: synchronize housekeeping
Dec 25 10:36:50 node-01 kubelet[1579]: I1225 10:36:50.324084    1579 helpers.go:836] eviction manager: observations: signal=nodefs.inodesFree, available: 2576153, capacity: 2621440, time: 2018-12-25 10:36:50.316134525 +0800 CST m=+5055.852141477
Dec 25 10:36:50 node-01 kubelet[1579]: I1225 10:36:50.324117    1579 helpers.go:836] eviction manager: observations: signal=imagefs.available, available: 35152652Ki, capacity: 41151808Ki, time: 2018-12-25 10:36:50.316134525 +0800 CST m=+5055.852141477
Dec 25 10:36:50 node-01 kubelet[1579]: I1225 10:36:50.324128    1579 helpers.go:836] eviction manager: observations: signal=imagefs.inodesFree, available: 2576153, capacity: 2621440, time: 2018-12-25 10:36:50.316134525 +0800 CST m=+5055.852141477
Dec 25 10:36:50 node-01 kubelet[1579]: I1225 10:36:50.324134    1579 helpers.go:836] eviction manager: observations: signal=pid.available, available: 32597, capacity: 32Ki, time: 2018-12-25 10:36:50.323700022 +0800 CST m=+5055.859706968
Dec 25 10:36:50 node-01 kubelet[1579]: I1225 10:36:50.324141    1579 helpers.go:836] eviction manager: observations: signal=memory.available, available: 502684Ki, capacity: 1016168Ki, time: 2018-12-25 10:36:50.316134525 +0800 CST m=+5055.852141477
Dec 25 10:36:50 node-01 kubelet[1579]: I1225 10:36:50.324148    1579 helpers.go:836] eviction manager: observations: signal=allocatableMemory.available, available: 1014508Ki, capacity: 1016168Ki, time: 2018-12-25 10:36:50.323993033 +0800 CST m=+5055.859999949
Dec 25 10:36:50 node-01 kubelet[1579]: I1225 10:36:50.324154    1579 helpers.go:836] eviction manager: observations: signal=nodefs.available, available: 35152652Ki, capacity: 41151808Ki, time: 2018-12-25 10:36:50.316134525 +0800 CST m=+5055.852141477
Dec 25 10:36:50 node-01 kubelet[1579]: I1225 10:36:50.324179    1579 eviction_manager.go:317] eviction manager: no resources are starved
Dec 25 10:36:50 node-01 kubelet[1579]: I1225 10:36:50.327740    1579 setters.go:736] Error getting volume limit for plugin kubernetes.io/azure-disk
Dec 25 10:36:50 node-01 kubelet[1579]: I1225 10:36:50.327750    1579 setters.go:736] Error getting volume limit for plugin kubernetes.io/gce-pd
Dec 25 10:36:50 node-01 kubelet[1579]: I1225 10:36:50.327754    1579 setters.go:736] Error getting volume limit for plugin kubernetes.io/aws-ebs
Dec 25 10:36:51 node-01 kubelet[1579]: I1225 10:36:51.195899    1579 kubelet.go:1995] SyncLoop (housekeeping)
Dec 25 10:36:51 node-01 kubelet[1579]: I1225 10:36:51.614440    1579 kubelet.go:2189] Container runtime status: Runtime Conditions: RuntimeReady=true reason: message:, NetworkReady=true reason: message:

kubectl describe pods my-web;

Events:
  Type    Reason     Age   From                     Message
  ----    ------     ----  ----                     -------
  Normal  Scheduled  13s   default-scheduler        Successfully assigned default/my-web to 172.16.127.119
  Normal  Pulling    12s   kubelet, 172.16.127.119  pulling image "nginx"
  Normal  Pulled     8s    kubelet, 172.16.127.119  Successfully pulled image "nginx"

[root@node-01 log]# docker image ls
REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
redis                                                             latest              ce25c7293564        11 days ago         95MB
nginx                                                             latest              568c4670fa80        3 weeks ago         109MB
registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64   3.0                 99e59f495ffa        2 years ago         747kB
-- user10830520
kubernetes

0 Answers