pods unable to get scheduled on node because of misleading resource overcommitted message k8s

8/19/2018

pod stuck in Waiting: PodInitializing state in Kubernetes though node has available resource..

Labels:             beta.kubernetes.io/arch=amd64

                    beta.kubernetes.io/os=linux

                    kubernetes.io/hostname=****

                    node-role.kubernetes.io/worker=true

Annotations:        node.alpha.kubernetes.io/ttl=0

                    volumes.kubernetes.io/controller-managed-attach-detach=true

Taints:             <none>

CreationTimestamp:  Tue, 31 Jul 2018 09:59:24 -0400

Conditions:

  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message

  ----             ------  -----------------                 ------------------                ------                       -------

  OutOfDisk        False   Tue, 14 Aug 2018 09:31:20 -0400   Tue, 31 Jul 2018 09:59:24 -0400   KubeletHasSufficientDisk     kubelet has sufficient disk space available

  MemoryPressure   False   Tue, 14 Aug 2018 09:31:20 -0400   Tue, 31 Jul 2018 09:59:24 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available

  DiskPressure     False   Tue, 14 Aug 2018 09:31:20 -0400   Tue, 31 Jul 2018 09:59:24 -0400   KubeletHasNoDiskPressure     kubelet has no disk pressure

  Ready            True    Tue, 14 Aug 2018 09:31:20 -0400   Sat, 11 Aug 2018 00:41:27 -0400   KubeletReady                 kubelet is posting ready status. AppArmor enabled

Addresses:

  InternalIP:  *****

  Hostname:    ******

Capacity:

 cpu:     16

 memory:  125827288Ki

 pods:    110

Allocatable:

 cpu:     16

 memory:  125724888Ki

 pods:    110

System Info:

 Machine ID:                 *************

 System UUID:                **************

 Boot ID:                    1493028d-0a80-4f2f-b0f1-48d9b8910e9f

 Kernel Version:             4.4.0-1062-aws

 OS Image:                   Ubuntu 16.04.4 LTS

 Operating System:           linux

 Architecture:               amd64

 Container Runtime Version:  docker://Unknown

 Kubelet Version:            v1.8.3

 Kube-Proxy Version:         v1.8.3

PodCIDR:                     ******

ExternalID:                  **************

Non-terminated Pods:         (11 in total)

  Namespace                  Name                                                            CPU Requests  CPU Limits  Memory Requests  Memory Limits

  ---------                  ----                                                            ------------  ----------  ---------------  -------------

  kube-system                calico-node-gj5mb                                               250m (1%)     0 (0%)      0 (0%)           0 (0%)

  kube-system                kube-proxy-****************************************             100m (0%)     0 (0%)      0 (0%)           0 (0%)

  kube-system                prometheus-prometheus-node-exporter-9cntq                       100m (0%)     200m (1%)   30Mi (0%)        50Mi (0%)

  logging                    elasticsearch-elasticsearch-data-69df997486-gqcwg               400m (2%)     1 (6%)      8Gi (6%)         16Gi (13%)

  logging                    fluentd-fluentd-elasticsearch-tj7nd                             200m (1%)     0 (0%)      612Mi (0%)       0 (0%)

  rook                       rook-agent-6jtzm                                                0 (0%)        0 (0%)      0 (0%)           0 (0%)

  rook                       rook-ceph-osd-10-6-42-250.accel.aws-cardda.cb4good.com-gwb8j    0 (0%)        0 (0%)      0 (0%)           0 (0%)

  spark                      accelerate-test-5-a3bfb8a597e83d459193a183e17f13b5-exec-1       2 (12%)       0 (0%)      10Gi (8%)        12Gi (10%)

  spark                      accelerate-testing-1-8ed0482f3bfb3c0a83da30bb7d433dff-exec-5    2 (12%)       0 (0%)      10Gi (8%)        12Gi (10%)

  spark                      accelerate-testing-2-8cecc18bb42f31a386c6304bd63e9eba-driver    1 (6%)        0 (0%)      2Gi (1%)         2432Mi (1%)

  spark                      accelerate-testing-2-e8bd0607cc693bc8ae25cc6dc300b2c7-driver    1 (6%)        0 (0%)      2Gi (1%)         2432Mi (1%)

Allocated resources:

  (Total limits may be over 100 percent, i.e., overcommitted.)

  CPU Requests  CPU Limits  Memory Requests  Memory Limits

  ------------  ----------  ---------------  -------------

  7050m (44%)   1200m (7%)  33410Mi (27%)    45874Mi (37%)


Events:         <none>
-- shiv455
kubernetes

0 Answers